WeLL System Status

Updated Thu Jun 15 11:58:00 PST 2017

If you make several attempts to login with the wrong password, the system may lock you out by blocking further connections from you. Unfortunately, automated brute-force password hacking bots make this lockout essential, so we cannot disable the processes that handle this.

For the conferences (SSH/Picospan and Engaged), a lockout presents as an inability to reach either well.com or user.well.com at all. In most cases, the lockout lasts about 3 hours, though we have temporarily reset it to 1 hour.

For email, lockouts may present in a variety of ways. Sometimes, your mail client will simply refuse to fetch or send mail, with some sort of cryptic message. Worse yet, some clients may mistakenly assume that you have the wrong password, and will ask you to enter the correct password. Don't try entering your password or creating a new password - this may make matters worse. Just be patient; the system will automatically unlock your account after about one hour.

Updated Tue Jun 06 21:38:00 PST 2017

If you attempt to log in and your password does not work, that means it was reset as a result of the outage on June 6th. Please visit https://www.well.com/cgi-bin/newpass/newpass.pl to reset your password. If you do not have an off-WELL email address on file, you will need to call Helpdesk at (415) 343-5731 and we'll help you reset your password. we'll be manning that phone from 9am-3pm PDT tomorrow.

Updated Tue Jun 06 12:04:00 PST 2017

As of approximately 11:30am PDT, we have taken The WELL down for unscheduled maintenance. We discovered a serious problem that requires taking the conferencing offline in order to fix. As of now, I don't have any additional information but will pass news along to you as I learn it.

We are estimating ~24 hours downtime - I'll post updates here as we get a better sense of how long we will be down.

We anticipate losing very few, if any, posts, and no user files, as a result of this issue.

As of now, mail (iris.well.com) is unaffected, and we do not anticipate that changing.

I will continue to update this page, as well as https://www.well.com/status.html, as I get more information.

Thank you for your patience!

Updated Wed Apr 12 17:25:00 PST 2017

All http://www.well.com URLs now redirect to https://www.well.com for security. See welltech.382, responses 1591 and up for details/problems.

Updated Tue Apr 04 20:56:00 PST 2017

We have started converting www.well.com to a purely https site in accordance with the requirements of recent versions of some browsers and Google's search priorities.

On Thursday, we will switch user web pages at www.well.com to use https. See welltech.382, responses 1591 and up.

Updated Tue Feb 21 07:26:00 PST 2017

Email upgrade postponed yet again. Probably next week

Updated Mon Feb 15 01:14:00 PST 2017

Email upgrade postponed until Tuesday night due to holiday.

The email system (iris.well.com aka Zimbra) will be upgraded on Tuesday evening Feb 21 starting about 9pm US Pacific Time. We expect to complete the upgrade by 5am Wednesday. Incoming mail will be queued locally, and delivered as soon as possible after the system is returned to service.

Updated Mon Feb 15 01:14:00 PST 2017

Email upgrade failed. We will try again next week.

The email system (iris.well.com aka Zimbra) will be upgraded on Monday evening Feb 20 starting about 9pm US Pacific Time. We expect to complete the upgrade by 5am Tuesday. Incoming mail will be queued locally, and delivered as soon as possible after the system is returned to service.

Updated Mon Feb 13 08:56:00 PST 2017

Email upgrade postponed until Tuesday night/Wednesday morning!

The email system (iris.well.com aka Zimbra) will be upgraded on Tuesday evening Feb 14 starting about 9pm US Pacific Time. We expect to complete the upgrade by 5am Wednesday. Incoming mail will be queued locally, and delivered as soon as possible after the system is returned to service.

Updated Fri Feb 10 12:47:00 PST 2017

The email system (iris.well.com aka Zimbra) will be upgraded on Monday evening Feb 13 starting about 9pm US Pacific Time. We expect to complete the upgrade by 5am Tuesday. Incoming mail will be queued locally, and delivered as soon as possible after the system is returned to service.

Updated Sun Jul 24 22:09:00 PST 2016

well.com/user.well.com should be back up -- investigation proceeding

Updated Sun Jul 24 21:59:00 PST 2016

well.com/user.well.com is down -- had to rebuild system, rebooting now

Updated Sun Jul 24 21:35:00 PST 2016

well.com/user.well.com is down -- waiting for system to shut down so I can restart

Updated Sun Jul 24 21:20:41 PST 2016

well.com/user.well.com is down -- investigating.

Updated Tue Jun 27 09:29:00 PST 2016

On Monday evening, 27 Jun, at 10pm US Pacific Time, there will be a brief downtime for well.com, user.well.com, and www.well.com; this includes all conferencing, alpine email, and web pages. We anticipate that the outage will be no more than 15 minutes in length, though we are scheduling a full hour, just in case.

Updated Tue May 24 09:33:26 PST 2016

The conferencing system is back and running after the upgrade.

If you find you cannot reach well.com or user.well.com, try rebooting your computer and all components of your network (routers, modems, wireless servers, etc). If this does not solve the problem, your DNS server may be slow about updating its cache. It will probably update in the next few hours. In the meantime, you can use the IP number 23.22.72.90 in place of "well.com" or "user.well.com". Note that your browser may issue dire warnings when accessing via the IP number.

If you use ssh, you will likely see a complaint that looks something like this the first time you connect to the new system:

  The authenticity of host 'well.com (23.22.72.90)' can't be
  established.
  RSA key fingerprint is

  xxxxxxxxxxxxxxxxxxxxx.

  Are you sure you want to continue connecting (yes/no)?

You may answer yes to this message.

If you want to confirm that you really are connecting to the new system, you can use the web conferencing system to see the necessary fingerprints. The fingerprints may be found HERE after you have logged in

If you see this:

  @       WARNING: POSSIBLE DNS SPOOFING DETECTED!          @
  @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
  The RSA host key for well.com has changed,
  and the key for the corresponding IP address 23.22.72.90
  is unknown. This could either mean that
  DNS SPOOFING is happening or the IP address for the host
  and its host key have changed at the same time.
  @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
  @    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
  @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
  IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
  Someone could be eavesdropping on you right now (man-in-the-middle
  attack)!
  It is also possible that a host key has just been changed.
  The fingerprint for the RSA key sent by the remote host is
  SHA256:bu4XkL/08dKLUmeT4JjhtenhAe4c3fV3yjSc1H6L9v0.
  Please contact your system administrator.
  Add correct host key in /Users/garylambert/.ssh/known_hosts to get
  rid of this message.
  Offending RSA key in /Users/garylambert/.ssh/known_hosts:1
  RSA host key for well.com has changed and you have requested strict
  checking.
  Host key verification failed.

Remove "well.com" from your "~/.ssh/known_hosts" file on your system.

UPDATE: Running this command from a Mac Terminal window seems to fix this:

ssh-keygen -f ~/.ssh/known_hosts -R well.com

Sun May 22 13:00:00 PST 2016

On Monday, May 23, starting at about 8pm US Pacific Time, the WELL's conferencing systems (well.com and user.well.com) will be unavailable due to system maintenance. The maintenance period is scheduled to run until about 4am Tuesday.

During the maintenance period, the operating system will be upgraded from CentOS 5.6 to CentOS 6.7. The upgrade is desirable from a standpoint of staying reasonably up to date, and for security reasons. All other systems (the main web server, the email servers, and the billing server) will all remain operational during the upgrade. However, the Alpine mail program will not be available.

When the system returns to operation, you may continue to have problems connecting for a short while due to DNS propagation issues. This should be rare - we have the propagation timeout set very, very low - but inevitably there are DNS systems that don't update their caches in a timely manner. We will post instructions on what to do in this case after we have turned the system back on. Until it becomes an issue, don't worry about it.

If you use our web interface, you should not see any differences after connecting.

If you use ssh, you will likely see a complaint that looks something like this the first time you connect to the new system:

  The authenticity of host 'well.com (23.22.72.90)' can't be
  established.
  RSA key fingerprint is

  xxxxxxxxxxxxxxxxxxxxx.

  Are you sure you want to continue connecting (yes/no)?

You may answer yes to this message.

If you want to confirm that you really are connecting to the new system, you can use the web conferencing system to see the necessary fingerprints. The fingerprints may be found HERE after you have logged in

Tue Apr 5 23:30:00 PST 2016

The WELL's primary web server, www.well.com, will be unavailable from 8pm US Pacific Time on Wednesday until around 2am on Thursday as we upgrade the operating system on this server. During the upgrade, WELL web pages at www.well.com will be unavailable, as will access to your WELL WEB folders. All other services will remain available throughout most of this upgrade.

Engaged - the WELL's browser-based conferencing - will be available throughout most of this upgrade. For those people who usually access Engaged by going to http://www.well.com, a very brief status page will be available - it will include a link to Engaged https://user.well.com/engaged.cgi.

The web server's IP number will change as part of this upgrade. This may result in complaints from your SFTP programs if you use one to manage your personal web pages. I think most browsers will be okay - they rely on the SSL certificate more than the specifics of the system. However, there is a chance that your browser might raise a security complaint.

At the very end of the upgrade, it may be necessary to reboot well.com and user.well.com - this should only take a few minutes if it happens, and will most likely be done after midnight.

Thu Feb 18 16:43:00 PST 2016

All seems well now.

Thu Feb 18 14:19:00 PST 2016

The system appears to have gotten stuck while rebooting. Investigating now.

Thu Feb 18 13:52:00 PST 2016

Re are experiencing issues with well.com temporarily losing connectivity, and are going to try a reboot in a few minutes. During the report, WELL conferencing (both shell access and Engaged) will be unavaiable.

Wed Dec 23 11:09:50 PST 2015

Mail is back, but it may take a while for your ISP to update the new DNS information.

Sat Dec 22 21:21:21 PST 2015

Thanks to unscheduled outage of the mail server (from about 8pm to 9:15pm), we managed to apply the patch originally scheduled for early tomorrow morning. So, that scheduled outage is canceled.

Sat Dec 22 14:23:31 PST 2015

We will apply a security patch to the Zimbra email software on Wednesday at about 1:30am US Pacific Time (immediately after the nightly backups). It shouldn't take long - we'll likely be back up around 2am.

Sat Dec 19 13:37:02 PST 2015

well.com and Engaged are back up and running. It looks like we just had an NFS software failure.

Sat Dec 19 13:12:28 PST 2015

well.com and Engaged have crashed. We are investigating and will have it back as soon as we can.

Sun Nov 22 11:52:47 PST 2015

The email system (iris.well.com) will be down briefly at about 6pm US Pacific Time to address the repeated crashes. The system should be back up no later than 6:15pm.

Tue Nov 17 11:09:45 PST 2015

All systems are up and running.

Mon Nov 16 10:23:20 PST 2015

Email will be unavailable from 1:30 am to 2:30 am PST, Tuesday, November 17th. We are applying an update patch to the Zimbra software. Mail will be spooled and held for delivery or for sending out after the downtime. We are sorry for any inconvenience.

Fri Jun 26 01:25:40 PDT 2015

There will be a brief outage of our primary web server, www.well.com, on Sunday morning Jun 28 between about 4am and 6am US Pacific Time. The outage should last no more than about 5-10 minutes.

Thu Jun 18 13:09:02 PDT 2015

It's come to our attention that Gmail is currently delaying most email from our customers, resulting in substantial slowness and maybe even bounces of mail going to Gmail accounts. We believe that forwarding spam is the root cause of these delays, and we have been forced to disable forwarding to Gmail for several email accounts. Those emails will remain on our mailserver for direct pickup by those customers.

Fortunately, there is an alternative way to get your WELL mail on Gmail if you wish to - for most purposes, it acts just like forwarding, including Gmail spam filtering, without the negative side effects on other WELL customers.

To read your WELL email vial Gmail, instead of forwarding, use Gmail's mail POP fetching function using the instructions in Settings. Log into Gmail and go to https://mail.google.com/mail/u/0/?pli=1#settings/accounts Use "Check mail from other accounts (using POP3):" to set it up. This provides the benefits of forwarding without the downsides.

Remember, all well.com email customers may get well.com mail directly by logging on at iris.well.com

We are sorry we had to take this emergency action. Please drop by the News conference system status topic if you want to discuss it.

Thu Jun 11 09:09:08 PDT 2015

All systems are up and available. The scheduled maintenance of June 10-11 is complete, and all email should be processed and delivered within a few hours. Note that the IP of the server had to change, and that if you are having trouble connecting to iris.well.com, helpdesk may be able to assist you. You will find more about the situation in the System Status topic in the News or Welltech conferences.

Mon Nov 17 13:44:00 PST 2014

All systems are up and available. The conference systems will be rebooted at around 11pm US Pacific Time on Mon, Nov 17. Service should be restored 5-10 minutes later.

Thu Nov 13 00:17:34 PST 2014

All systems are up and available. The scheduled maintenance on Nov 12 is complete.

Wed Nov 12 15:04:04 PST 2014

All systems are up and available.

Conferencing via well.com and user.well.com will be unavailable from 10pm-midnight US Pacific Time on Wednesday, Nov 12. We apologize for the inconvenience.

Tue Nov 06 18:30:30 PDT 2014

Investigating system outage on well.com/user.well.com

Tue Oct 28 13:13:23 PDT 2014

All systems are up and available.

SECURITY UPDATE

Sometime on Wednesday, Oct 29, we will update the SSL certificate associated with iris.well.com (the WELL's email server, aka Zimbra, aka Iris) to one that uses the SHA-256 (aka SHA-2) hashing algorithm. The old SHA-1 algorithm has been deprecated, and many sources, including our certificate provider, are urging that the old certificates be replaced as soon as possible. (SSL certificates are used by browsers to ensure that the site you are accessing is the site that it claims to be.)

Note that this change could have incompatibility impacts on older browsers and email software. Please see welltech.374.920 for further details.

Sun Oct 19 02:41:13 PDT 2014

All systems are up and available.

At about 11:30pm on Oct 18, there was a configuration glitch that temporarily caused email to be unavailable due to apparent password problems. The glitch was fixed and service restored at about 2:30am. If you tried to change your password during this period, you may find that you now have two different passwords - one for mail, and one for everything else. We recommend updating your password once more to get them back in synch.

Fri Oct 10 10:15:24 PDT 2014

On Monday Oct 13 (Columbus Day) at 9:00 am PDT, the server at well.com (aka user.well.com) will be shut down for maintenance by Amazon Web Services. The maintenance is scheduled to last two hours, though we expect it will be much shorter than that.

During the maintenance period, the UNIX and web interfaces to our conferences, Picospan and Engaged, will not be available. The UNIX shells as well as the Alpine mail program will also be unavailable. However, email will still be available at iris.well.com. Both sending and receiving email will also continue to be available via POP and IMAP, as usual. The www.well.com website will also remain up.

We apologize for any inconvenience this maintenance presents.

Wed Sep 10 17:06:24 PDT 2014

A major update to our web conferencing software, commonly referred to as "Engaged," has been completed. To report any new bugs or to read the scope of the updates please see the System Status Reports topic in News or WellTech.

Some highlights:

For more, including updates to parsing and formatting, please see the System Status Reports topic in News or WellTech. Questions and bug reports are welcome.

All systems remain up and available.

Tue Apr 15 08:06:24 PDT 2014

All systems are up and available.

Tue Apr 8 13:19:51 PDT 2014

The email system received an urgent security update today. It is possible that your browser or mail program will complain about not being able to verify the identity of iris.well.com. Please see welltech.374.334- for info.

Tue Oct 1 14:54:53 PDT 2013

Planned Outage: Email Maintenance and Upgrade Rescheduled for 10/2 - 10/3

03:08 POP access is now available: This completes the Zimbra 8 upgrade: Please see the welltech conference for information about the update, and to report any issues.
03:00 IMAP and Alpine access is now available: you can send and read mail via your IMAP client or the Apline command-line interface. Things will continue to be slow for a while.
02:50 SMTP service is available: you can send mail via your IMAP or POP client, and newly arrived email is being delivered to your mailboxes. Forwarding should also be working at this time.
02:41 Web access to your email is available: you can send and read mail via your browser. Things will be slow for a couple of hours, but they should be working.
01:54 Upgrade complete, starting system
System will be started one component at a time, and may take some time to become completely available.
23:51 Zimbra 8 installed, now configuring.
21:54 Installing Zimbra 8.
21:54 Preparing system for Zimbra 8 update.
20:00 Initial setup for upgrade is underway.

There will be no outbound well.com email service -- and no access to inbound mail -- starting Wednesday 10/2 at 8pm US Pacific Time. while we upgrade the email system to the latest version, Zimbra 8. We expect to restore services at 4am Pacific Time, Thursday, 10/2.

Incoming mail will safely queue on the server until we are back. During the outage, you will not be able to send, receive, or view mail, nor can it forward.

We apologize for any inconvenience. For updates, please watch this page. For discussion, go to the System Status discussions in the News or WELLtech conferences.

Mon Sep 30 9:20:39 PDT 2013

Access has been restored. The upgrade will be rescheduled, due to circumstances beyond our control.

Log in and see the System Status topic in the News or WELLtech conferences if you want to get the details or ask questions.

Mail was queued and then delivered after the system came back up. Sorry for the inconvenience.

Sun Sep 29 22:52:39 PDT 2013

Update: Access will be restored ASAP, due to circumstances beyond our control.

This may be before or after the original target time of 4am. We do not have a new estimated time yet. Log in and see the System Status topic in the News or WELLtech conferences for additional details.

We apologize for any inconvenience.

Sun Sep 29 20:31:22 PDT 2013

Planned Outage: Email Maintenance and Upgrade 9/29 - 9/30

Initiial setup for upgrade is underway. Expect an update to this page at about 10:30pm US Pacific Time.

There will be no outbound well.com email service -- and no access to inbound mail -- starting Sunday 9/29 at 8pm US Pacific Time. while we upgrade the email system to the latest version, Zimbra 8. We expect to restore services at 4am Pacific Time, Monday, 9/30.

Incoming mail will safely queue on the server until we are back. During the outage, you will not be able to send, receive, or view mail, nor can it forward.

We apologize for any inconvenience. For updates, please watch this page. For discussion, go to the System Status discussions in the News or WELLtech conferences.

Tue Apr 9 22:14:46 PST 2013

All systems are running normally.