Author Topic: lendacademy web server problems  (Read 37796 times)

anabio

  • Full Member
  • ***
  • Posts: 109
    • View Profile
Re: lendacademy web server problems
« Reply #45 on: November 01, 2016, 03:49:56 PM »
Hi all, just an update from our end. Our hosting provider isn't being very helpful resolving this issue so we are going to move to another hosting provider that is optimized for our current forum software. We'll be working on this over the next week and will post any updates as well as any planned downtime here. Thanks for your patience.
Just a note of sympathy for both you  and your provider. I used to do that "stuff" before I retired and sometimes all you can do  is pull your hair out. Intermittent stuff like this is like looking for a needle in a haystack.  :-\ Give me a complete failure any day.
As Will Rogers stated: : I'm not as concerned about the return on my money as I am the return of my money

Fred93

  • Hero Member
  • *****
  • Posts: 2246
    • View Profile
Re: lendacademy web server problems
« Reply #46 on: November 04, 2016, 12:05:27 AM »
These events are not random.  They are HOURLY, plus or minus a few minutes.  In a few hours, after I've captured a few more events, I will post a chart showing the regularity.

Is there something that runs once per hour?

nonattender

  • Hero Member
  • *****
  • Posts: 726
  • I am not here.
    • View Profile
Re: lendacademy web server problems
« Reply #47 on: November 04, 2016, 12:52:39 AM »
These events are not random.  They are HOURLY, plus or minus a few minutes.  In a few hours, after I've captured a few more events, I will post a chart showing the regularity.

Is there something that runs once per hour?

It's probably fixed, but now being taken down once per hour by all of the nerds on this forum pinging it to do time-series analysis... ;)

I still say it's the Russians - and if you don't believe me (and vote for me) I'll send you and your charts and logic straight to the gulag.

ETA:

Here, go play with this - at least you might make some money for your time:

https://www.predictit.org/Home/Election2016

As best I can tell, the guy(?) running multiple accounts to "clear" the swing state markets for Hillary gets up and goes wild with his scripts at about 6:45am eastern.  Good luck.  His schedule may have changed a bit, now that things are tightening up against him.

Nothing says "uncorruptible wisdom of the crowds, purely for academic research" like a 50/50 proposition for ~.30 each morning. ;)

Or maybe it's just a whole bunch of silly people acting irrationally within an extremely small (~5 second) time window - who knows?

::)
« Last Edit: November 04, 2016, 01:54:30 AM by nonattender »
A little nonsense now and then is relished by the wisest men.

Fred93

  • Hero Member
  • *****
  • Posts: 2246
    • View Profile
Re: lendacademy web server problems
« Reply #48 on: November 04, 2016, 02:31:52 AM »
I wrote a program to periodically fetch the lendacademy main forum page, once per minute, and record how long this took, and whether an error was returned by the server.  The following chart shows the time-to-fetch the page.  The points at 60 seconds, and some around 30 seconds are the ones that report the 404 errors.  You can see that the events are not random, but are roughly periodic, with a period slightly under one hour.  If these events were random, you would sometimes see a pair close together, but that never happens.


Fred93

  • Hero Member
  • *****
  • Posts: 2246
    • View Profile
Re: lendacademy web server problems
« Reply #49 on: November 04, 2016, 06:19:32 AM »
Woo hoo.  Has now run for a few more hours, and the periodic nature of the problem could not be more clear.  I've attached an updated chart where you see 10 evenly spaced events, ie 9 periods complete in 8 hours.  Although each event has an uncertainty of about 5 minutes, the cycle is 55 minutes long.

Very few malfunctions would cause something like this.

If you have a cron job or other periodic thing doing something at near this interval, look there. 

If you have a back end database server, I would say it is likely restarting every 55 minutes, hence is unavailable to the web server for a couple of minutes.  What could make a back end server restart periodically?  Some software with a memory leak could do it.  After about 55 minutes of operation, memory is used up, and the database server restarts.

Can you examine logs to see if the database server, or some similar component, is restarting?  Can you monitor memory on the backend server to see if free memory is steadily decreasing during the 55 minute interval between events?

I think it is unlikely that this is a hardware problem.  I can imagine a hardware problem that occurs after X minutes of operation, such as thermal problem for example, but it is difficult to imagine that it gets fixed by a quick reset of something.  Once something is too hot would need time to cool.

nonattender

  • Hero Member
  • *****
  • Posts: 726
  • I am not here.
    • View Profile
Re: lendacademy web server problems
« Reply #50 on: November 04, 2016, 06:27:18 AM »
Ok, Fred...  it's been 10 years... would you please, for the love of all things Holy, go take some kind of minister-without-portfolio consulting gig at one of the platforms where you're tasked with just walking around the joint and finding inoptimal/broken shit - and then fixing it? :)

I'd so invest.
A little nonsense now and then is relished by the wisest men.

RyanL

  • Administrator
  • Newbie
  • *****
  • Posts: 27
    • View Profile
Re: lendacademy web server problems
« Reply #51 on: November 04, 2016, 10:33:59 AM »
Fred, appreciate your analysis - unfortunately I only have access to one set of logs in which there is a consistent apc_store error. The host has said they've looked and have found no issues with APC on the server. They keep coming back to: "Most, if not all of the current processes for the day are queried URLs and as we explained, this bypasses caching." So they are blaming caching on the issue (keep in mind we have made no changes and visits to the site have not substantially increased). At the end of the day, they are not optimized for SMF forums and they don't provide adequate support either.

11/3/2016 11:20:35 AM - www: PHP Warning: apc_store(): GC cache entry 'f5a7014e25789cff0106c3999bf7c344-SMF-modSettingssmf' was on gc-list for 3602 seconds in /var/www/lendacademy.com/forum/Sources/Load.php on line 2687

Fred93

  • Hero Member
  • *****
  • Posts: 2246
    • View Profile
Re: lendacademy web server problems
« Reply #52 on: November 04, 2016, 03:20:19 PM »
11/3/2016 11:20:35 AM - www: PHP Warning: apc_store(): GC cache entry 'f5a7014e25789cff0106c3999bf7c344-SMF-modSettingssmf' was on gc-list for 3602 seconds in /var/www/lendacademy.com/forum/Sources/Load.php on line 2687

Do these messages occur at times that are sync'd to the events?  If not, might be meaningless.  Lots of people get these messages apparently, as there are many google hits. 

Are you running latest versions of PHP and APC?  Comments on web say this message was changed in 2013.  Maybe you are stuck running an old version because the web hosting guys are way behind? 

http://stackoverflow.com/questions/34725669/apcu-configuration-gc-ttl-0/34726006
"Which means you must be on old APC version and it is better to upgrade."
https://github.com/krakjoe/apcu/pull/45

Have you tried disabling the APC cache?  It might be the bad guy who is running out of memory and restarting.  Or maybe not.
https://coolicehost.com/billing/knowledgebase.php?action=displayarticle&id=12

In any case, good luck with the hosting transition.  Hope you are able to figure out how to do it without the typical 2 day downtime due to DNS.  If you think we go nuts when LA is down for 3 minutes, imagine what will happen if we have a 2 day outage!  I feel sure that it is possible.  I figure if you keep the old IP address, and set it up to redirect HTTP to the new IP address, tho I've never worked out the details.  The thing to remember is that different users will get different IP addresses from DNS during the transition, which makes the transition a difficult thing to test.

wiseclerk_com

  • Newbie
  • *
  • Posts: 43
    • View Profile
    • P2P-Banking.com
Re: lendacademy web server problems
« Reply #53 on: November 05, 2016, 07:32:35 AM »
extremly slow response in the past 5 minutes
I publish the http://www.p2p-banking.com blog

RyanL

  • Administrator
  • Newbie
  • *****
  • Posts: 27
    • View Profile
Re: lendacademy web server problems
« Reply #54 on: November 10, 2016, 05:37:36 PM »
Hi all, please note that the forum will be moving to our new host on Friday November 11th at 8 pm PST. At this time the forum will be unavailable for *hopefully* just 7 hours.

-Ryan

Rob L

  • Hero Member
  • *****
  • Posts: 2133
    • View Profile
Re: lendacademy web server problems
« Reply #55 on: November 10, 2016, 06:25:56 PM »
Good move! We'll look forward to business as usual thereafter. It  has bee tedious.

Fred93

  • Hero Member
  • *****
  • Posts: 2246
    • View Profile
Re: lendacademy web server problems
« Reply #56 on: November 10, 2016, 06:34:21 PM »
Hi all, please note that the forum will be moving to our new host on Friday November 11th at 8 pm PST. At this time the forum will be unavailable for *hopefully* just 7 hours.

but...
Quote
Got answer:
    HEADER:
        opcode = QUERY, id = 27, rcode = NOERROR
        header flags:  response, want recursion, recursion avail.
        questions = 1,  answers = 1,  authority records = 1,  additional = 0

    QUESTIONS:
        www.lendacademy.com, type = AAAA, class = IN
    ANSWERS:
    ->  www.lendacademy.com
        canonical name = lendacademy.com
        ttl = 2933 (48 mins 53 secs)
    AUTHORITY RECORDS:
    ->  lendacademy.com
        ttl = 600 (10 mins)
        primary name server = pdns03.domaincontrol.com
        responsible mail addr = dns.jomax.net
        serial  = 2016100300
        refresh = 28800 (8 hours)
        retry   = 7200 (2 hours)
        expire  = 604800 (7 days)
        default TTL = 600 (10 mins)

If your DNS record is set to expire in 7 days, doesn't that mean some DNS servers in the chain will hold the old info for several days?  Don't you need to change the expire to something much shorter, and do that several days before the move?   ...or am I misreading something here?

RyanL

  • Administrator
  • Newbie
  • *****
  • Posts: 27
    • View Profile
Re: lendacademy web server problems
« Reply #57 on: November 11, 2016, 03:55:00 PM »
Hi all, please note that the forum will be moving to our new host on Friday November 11th at 8 pm PST. At this time the forum will be unavailable for *hopefully* just 7 hours.

but...
Quote
Got answer:
    HEADER:
        opcode = QUERY, id = 27, rcode = NOERROR
        header flags:  response, want recursion, recursion avail.
        questions = 1,  answers = 1,  authority records = 1,  additional = 0

    QUESTIONS:
        www.lendacademy.com, type = AAAA, class = IN
    ANSWERS:
    ->  www.lendacademy.com
        canonical name = lendacademy.com
        ttl = 2933 (48 mins 53 secs)
    AUTHORITY RECORDS:
    ->  lendacademy.com
        ttl = 600 (10 mins)
        primary name server = pdns03.domaincontrol.com
        responsible mail addr = dns.jomax.net
        serial  = 2016100300
        refresh = 28800 (8 hours)
        retry   = 7200 (2 hours)
        expire  = 604800 (7 days)
        default TTL = 600 (10 mins)

If your DNS record is set to expire in 7 days, doesn't that mean some DNS servers in the chain will hold the old info for several days?  Don't you need to change the expire to something much shorter, and do that several days before the move?   ...or am I misreading something here?

I was told by our new host that since we are not moving over the main domain that servers the WP it will be fine. Just a reminder, the forum will be down starting at 8 pm PDT tonight. The new url will also be forum.lendacademy.com and we will make sure all current pages redirect correctly.

Fred93

  • Hero Member
  • *****
  • Posts: 2246
    • View Profile
Re: lendacademy web server problems
« Reply #58 on: November 11, 2016, 04:54:59 PM »
I was told by our new host that since we are not moving over the main domain that servers the WP it will be fine.

Oh, that sounds like it will work.  With no old domain changing IP, the DNS caching issues are mute.

I've never seen anybody get a transition like this exactly right.  May you be the first!

rawraw

  • Hero Member
  • *****
  • Posts: 2795
    • View Profile
Re: lendacademy web server problems
« Reply #59 on: November 11, 2016, 05:05:34 PM »
Fred93 cracks me up.  This thread was awesome