Fri Apr 5 13:09:45 PDT 2013
There will be a brief outage of well.com (the shell/picospan host), user.well.com (web conferencing/Engaged), and www.well.com (the web server) Monday at 10pm US Pacific Time. Each system needs to be rebooted so we can enable controls to prevent disk overruns. All systems should be back and running by 10:30pm.
Tue Feb 26 22:54:49 PST 2013
All systems are running normally.
Tue Feb 26 12:26:10 PST 2013
Mail is currently unavailable as we clean up from last week's upgrade. Mail is expected to return by midnight tonight. See you then!
Thu Feb 21 06:09:16 PST 2013
Last night's mail system upgrade appears to have been successful. However, it took 4 hours longer than we scheduled for due to unforeseen issues with the Amazon "Cloud". See welltech.370 for details of what's new.
Wed Feb 20 20:00:00 PDT 2013
Mail is currently unavailable as a software upgrade is performed. Mail is expected to be available by 2am Thursday morning. See you then!
Update as of 02:47:00: we're running way behind schedule. Currently, we're expecting to be back online at around 4:30am. We apologize for the inconvenience, and especially the longer than expected outage.
Mon Jan 7 14:48:22 PST 2013
Due to an NFS issue, user WEB directories and short-term backup recovery snapshots are unavilable on well.com. User web pages are still available at http://www.well.com, and files can be updated with scp or sftp at www.well.com, but files cannot be viewed or updated from the well.com host.
In addition, you may see problems when trying to login or logout from the Web-based conferencing software (Engaged). To login, go to http://www.well.com. The error on logout can be ignored.
Mon Dec 3 16:32:55 PST 2012
A system reboot is scheduled for Tuesday, Dec 4, 11pm US Pacific Time. The reboot will last about 5 minutes, and will effect shell and conferencing access (both via SSH and the Web).
Wed Nov 21 22:31:45 PST 2012
The restart of our SMTP server is taking longer than anticipated. We have an open ticket with Amazon Web Services to investigate why the system is not shutting down properly. We don't yet have an ETA.
Wed Nov 21 17:10:13 PST 2012
Amazon has alerted us that our SMTP server is running on "degraded hardware". To get things back to normal, we will be rebooting the server later tonight (around 10pm). If all goes well, it should only take about 5 minutes.
Alpine users are most likely to see any effects - while the server is rebooting, Alpine will complain if you try to send email. Just resend the mail a few minutes later if this happens to you. Other customers may seem some minor delays on mail being received or sent, but in almost all cases, the delayed mail should be delivered properly some time after the system comes back up.
Sorry for any inconvenience this reboot will cause...
Tue Oct 2 22:03:27 PDT 2012
All systems are up and running.
However, we have changed our DNS authoritative server, and some DNS servers are being slow to pick up this change. If you are having trouble connecting to any WELL machines, you may want to change your DNS server on your router, Mac, or PC to use 22.214.171.124 as the primary server. However, be careful to keep a record of the previous settings, since you will likely want to change these settings back at some point. Your best bet is to wait until your normal DNS server picks up the updates, which should happen within a few hours in most cases.
NOTE: It looks like there have been a few cases of mail bouncing from about 7:40pm-8:50pm, and again from about 10:05pm-10:20pm US Pacific Time. Please ask senders to resend if you are missing mail from this period.
We apologize for any inconvenience resulting from this necessary update.
Fri Jun 15 12:18:22 PDT 2012
Amazon EC2 services, the cloud where our servers reside, has called for us to shut down and relaunch well.com and user.well.com, the servers where WELL conferencing lives. Tomorrow, Saturday June 16, those services will be unavailable from approximately 9pm until 11:00pm Pacific Daylight Time while wolfy migrates services to new instances. (Last night there was significant disruption that was probably unrelated to this next action, but which also came from problems at the Amazon cloud. Sorry for the inconvenience.) Thank you for your patience and loyalty.
Fri Jun 15 09:38:22 PDT 2012
All systems are up and running.
Fri Jun 15 01:22:04 PDT 2012
Email is down due to problems in the Amazon Cloud. We do not have an ETA at this time, but email may not be available until late morning or later.
Wed Mar 14 15:04:28 PDT 2012
We will be rebooting well.com and user.well.com Wednesday evening (Mar 14) at about 11pm US Pacific Time to make a minor configuration change. The reboot and maintenance will take about 20 minutes, and will effect access to the conferences via both Picospan and the Web, as well as all shell services (including alpine mail). We apologize for the inconvenience.
Thu Mar 1 22:26:11 PST 2012
(ETA changed to midnight - we're running a bit behind schedule)
On Thursday evening between about 9pm and midnight US Pacific Time, WELL conferencing via UNIX shell and the web will be down, with all access to well.com and user.well.com disabled for the duration. During this maintenance, we will be switching over to a new system kernel - we expect that this will put an end to the kernel panics we have seen over the last two months.
Access to your email via https://iris.well.com/, POP, and IMAP will not be affected by this maintenance. However, please note that access to email via the "alpine" mail program will not be available since well.com itself will be unavailable.
We apologize for the inconvenience of this maintenance, but random system crashes are even more inconvenient.
Thu Feb 23 21:15:19 PST 2012
ENGAGED IS DOWN - the database was accidentally trashed. We hope to have it back up by 10:30pm US Pacific Time.
Wed Jan 11 17:00:35 PST 2012
We did not reboot Tuesday, so we will be rebooting well.com tonight, Jan 11, at about 10pm US Pacific Time. The reboot will take 5 to 10 minutes, and will effect all uses of the conferences.
Wed Jan 11 09:37:59 PST 2012
All systems are up and running.
Tue Jan 10 12:19:25 PST 2012
We will be rebooting well.com tonight at about 10pm US Pacific Time in an effort to address the repeated system panics. The reboot will take 5 to 10 minutes, and will effect all uses of the conferences.
Fri Dec 16 15:43:25 PST 2011
After a reboot, people are back posting. All systems are up and running.
Fri Dec 16 14:55:25 PST 2011
We are having a problem with the well.com and user.well.com machines. While we resolve this problem, the conferences -- our forums -- are unreachable. Email at iris.well.com and the www.well.com web page remain up. We are sorry for the inconvenience.
Fri Dec 9 10:53:25 PST 2011
All systems are up and running.
Thu Dec 8 2011 16:57:24 PST 2011
Our service provider, Amazon, needs to reboot our systems to perform some sort of system update (presumably patches). Fortunately, we can prevent the automatic reboot by rebooting manually. The best time for us to do this is tonight, even though that means a short notice outage.
So, tonight at about 10pm US Pacific Time, each of the various well.com systems will be brought down then rebooted. Each reboot should take less than 5 minutes, though the reboot of the Zimbra machine may take a few minutes longer (it takes several minutes just for Zimbra to shut down, and another few minutes to start up).
We apologize for the short notice for this announcement.
Thu Oct 27 14:10:50 PDT 2011
Most remaining bugs appear to have been eradicated. If you encounter any, please report them in the cloud discussion in the welltech conference. Log in, shortcut is "g welltech" to get there.
If you use the PicoSpan interface and you're getting error messages on login, the problem may be configuration in your personal log in files. If you send a message to helpdesk, we can back up and replace your current log in files with basic files so that you can test each line to find the culprit.
Wed Oct 26 07:45:03 PDT 2011
The WELL is now back up and running in the Amazon Elastic Computing Cloud (EC2). Please report any bugs by going to the cloud discussion in the welltech conference. Log in, shortcut is "g welltech" to get there. If there is a malformed conference mentioned in an error message on your your Conference List page when you first connect to open a page at "https://user.well.com/engaged.cgi?" you may want to remove the bad conference temporarily by editing your Settings. Reporting a bug could win you one of a bunch of interesting books, so don't hesitate.
Mon Oct 24 16:42:39 PDT 2011
Migration down-time planned
On Tuesday night, Oct 25, we finish up our migration into the Amazon Elastic Computing Cloud (EC2). The last machines to be moved will be the main machine (well.com), the conferencing web server (user.well.com), and the web page server (www.well.com).
As part of this migration, conferencing (via both the command line and web browser), as well as all access to The WELL's web pages, will be disabled. Email will remain available, except for the "alpine" mailer. We're anticipating that the migration should be complete in about 6 hours: from 9pm PDT Tuesday evening to 3am PDT Wednesday morning.
Following the migration, you should be able to use conferencing and all web services as before, with only minor changes that most people will not see. If by some chance you do experience problems after the system is restored to service, don't try to change your settings please. If we are still sorting out the move that could make things harder to diagnose. Please just contact helpdesk with any bug reports or post in the System Status topic in the WELL tech or News conference, so a proper diagnosis can be made.
We apologize for any inconvenience this downtime may cause.
Tues Aug 30 13:07:22 PDT 2011
Performance of the email system remains sluggish - we are working to restore good service.We apologize for any inconvenience this slowness may cause.
Fri Aug 26 13:43:56 PDT 2011
On Monday night, Aug 29, we begin our migration into the Amazon Elastic Computing Cloud (EC2). The first machines to be moved will be the Zimbra mail server (iris.well.com) and a companion server.
As part of this migration, email will be disabled while the migration is under way. We're anticipating that the migration should be complete in about 6 hours: from 9pm PDT Monday evening to 3am PDT Tuesday morning.
During the migration, incoming email from other sites should be queued at those sites, and re-sent later after we begin accepting incoming email again. WELL customers will not be able to look at previously received email during the outage unless you have already moved that email onto your computer's storage device.
Following the migration, you should be able to continue using email using the settings that have worked for the last few weeks - nothing should change. If you encounter problems after the migration, please avoid changing your mail settings and report the problems to Helpdesk or post in System Status Reports.We apologize for any inconvenience this downtime may cause.
Thu Aug 18 21:37:03 PDT 2011
The WELL is now back up and running. Engaged is still presenting some problems, but for now, you can make your own repairs using the instructions on your Conference List page.
Thu Aug 18 14:54:03 PDT 2011
RAID failure knocked out some services, currently being restored
Around 1:00 am on the morning of Thursday, August 18th, home directories became unreachable at The WELL, causing all services except email to fail.
At 9:30 the following morning we we were able to place the following unadorned message on the well.com site:
Most functions of The WELL are currently not available due to a hardware failure of the RAID where most of our data is kept. We are working on restoring full service, but do not yet have an estimated completion time. We apologize for this inconvenience.
Please note that email via IMAP, POP, and the Web should be available at this time.
During that time we went through some replacement hardware options, settled on one, and got it installed at our co-location facility. At about 14:45 PDT logins became possible again.
RAIDs are designed to prevent such problems when a disk fails, and we are in the process of moving well.com to the cloud, so this failure was perplexing and hopefully very soon all questions will become moot. Some services are still being repaired and adjusted at this time. Comments are welcome, please email Helpdesk at the address on the footer of this page, or swing by the welltech conference.
We are sorry for the disruption.
Mon Aug 1 16:42:55 PDT 2011
Mail upgrade complete: If you can't get mail you need to change settings
All systems are up and running. We now have IMAP available for all mail customers. Please be advised that we have shut off all access to two older mail servers as of this afternoon as part of the upgrade to email started in 2009. If you are pointing to those servers in your settings, your mail will no longer work.
If you are not getting or sending email, you probably need to update some settings. You no longer have access to smtp.well.com nor to mh.well.com (mail.well.com). All incoming and outgoing mail operations will now involve the iris.well.com server. To get the appropriate Ports, look up your appropriate settings.
1. You should have access to your new email via a direct web browser login at iris.well.com now and henceforth. This is the easy, quick way for most people to see mail.
2. New mail help for all kinds of programs is now up at http://www.well.com/emailhelp.html with an overview of the system, general settings and a link to a page with more settings for specific mobile devices and computer email programs.
3. You can move your old email into the new environment by use of the utilities page if you wish.
4. There is a way to use Alpine for UNIX access to your mail. See the help page first.
5. We attempted to migrate you with settings matching your former preferences. In some situations we were not able to do that! In particular, if you were forwarding elsewhere and leaving a copy you are probably set up with straight forwarding now, and you will want to adjust that with a web login to iris.well.com.
If you have questions, contact helpdesk or ask fellow users who have solved the problem in the welltech conference. Thank you for your patience.
Mon Aug 1 09:59:46 PDT 2011
Mail upgrade is in progress.
All remaining mailboxes at The WELL are being transferred to the Zimbra email system at iris.well.com over the next several hours today. Please see the mail help page for instructions on setting up once your mailbox is converted, or use your web browser to go to iris.well.com. (Note that you cannot log in there until the upgrade script converts your mail settings.) Thank you for your patience.
Wed Jun 22 12:55:09 PDT 2011
All systems are up and running.
Monday, June 20, 2011 05:36:27 PM PDT
The Zimbra email system on iris.well.com will be unavailable Tuesday evening (Jun 21) between the hours of 10pm and midnight so that we can perform a required software update. The legacy email system will remain up during this period. Sorry for the inconvenience.
Fri May 13 13:50:40 PDT 2011
Members using the Legacy email server are finding their connections time out when they are trying to access mail from mail.well.com. We are working on the situation.