[collectd] apache: curl_easy_perform failed: Failed writing body

[kcheek at umich.edu ()]

First, thanks for creating collectd - I just stumbled across it this
weekend while looking for some tools to monitor linux vservers. It
looks like it might be very useful here.

Unfortunately, I'm having some trouble figuring out what I'm doing
wrong with the apache plugin. I've installed collectd on several
debian systems. A few are running sarge with collectd packages from
backports.org (3.10.1) and the others are running sid or etch with
collectd (3.10.3) debian packages. The systems are all configured as
clients and two are also servers (one sarge system and one sid system)
using multicast. Most plugins I've tried seem to work, but none of
the systems can successfully use the apache collectd plugin. No rrd's
are created for apache stats and I'm seeing the following syslog
messages every ten seconds:

collectd[9116]: apache: curl_easy_perform failed: Failed writing body

Per the FAQ, I checked to make sure apache and collectd are configured
to talk to each other properly.

All of the systems have the collectd-apache debian package installed
and apache is configured to allow status requests from localhost with
no username or password.

/etc/collectd/collectd.conf contains:

<Plugin apache>
URL http://localhost/server-status?auto
</Plugin>

The apache logs seem to indicate that the requests are successful:

127.0.0.1 - - [25/Nov/2006:15:15:44 -0500] "GET /server-status?auto HTTP/1.1" 200 4289 "-" "collectd/3.10.3" "-"

"curl http://localhost/server-status?auto" on the command line
produces:

Total Accesses: 682
Total kBytes: 2855
CPULoad: .0273771
Uptime: 6794
ReqPerSec: .100383
BytesPerSec: 430.309
BytesPerReq: 4286.69
BusyServers: 1
IdleServers: 4
Scoreboard: __W__.................................................................................................... ..................................................................................................................... ..................................................................................................................... ..................................................................................................................... ..................................................................................................................... ..................................................................................................................... ..................................................................................................................... ..................................................................................................................... ..................................................................................................................... ..................................................................................................................... ..................................................................................................................... ..................................................................................................................... ..................................................................................................................... ..................................................................................................................... ..................................................................................................................... ..................................................................................................................... ..................................................................................................................... ..................................................................................................................... ..................................................................................................................... ..................................................................................................................... ..................................................................................................................... ..................................................................................................................... ..................................................................................................................... ..................................................................................................................... ..................................................................................................................... ..................................................................................................................... ..................................................................................................................... ..................................................................................................................... ..................................................................................................................... ..................................................................................................................... ..................................................................................................................... ..................................................................................................................... ..................................................................................................................... ..................................................................................................................... ..................................................................................................................... .............


So far, I'm also unable to get statistics with the vserver or mysql
plugins, but I'm not seeing any error messages in syslog for those -
they just don't produce any rrd files.

The mysql.log shows messages like "061125 20:32:10 7 Query SHOW GLOBAL
STATUS" every ten seconds, so I'm guessing that collectd is able to
talk to mysqld.

I'm not sure where to look for problems with the vserver plugin.
/proc/virtual exists and it contains directories with status
information for each running vserver.

Any suggestions?

Thanks!

-kevin

[Sebastian Harl]

Hi Kevin,

Post by kcheek at umich.edu ()
I'm not sure where to look for problems with the vserver plugin.
/proc/virtual exists and it contains directories with status
information for each running vserver.

Uh... this is strange. The only thing that is not reported to syslog is an
error in a snprintf() call. I think this is pretty unlikely to happen as I'm
using a 512 bytes buffer to write to ;-)

Which version of Linux-VServer do you use? Could you please provide the
contents of the files cacct, cvirt and limit of some /proc/virtual/<XID>
directory?

Cheers,
Sebastian
--
Sebastian "tokkee" Harl
GnuPG-ID: 0x8501C7FC
http://tokkee.org/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
Url : http://mailman.verplant.org/pipermail/collectd/attachments/20061126/313c599a/attachment.pgp

[kcheek at umich.edu ()]

Post by Sebastian Harl
Hi Kevin,

Post by kcheek at umich.edu ()
I'm not sure where to look for problems with the vserver plugin.
/proc/virtual exists and it contains directories with status
information for each running vserver.

Uh... this is strange. The only thing that is not reported to syslog is an
error in a snprintf() call. I think this is pretty unlikely to happen as I'm
using a 512 bytes buffer to write to ;-)
Which version of Linux-VServer do you use? Could you please provide the
contents of the files cacct, cvirt and limit of some /proc/virtual/<XID>
directory?

Hi Sebastian, thanks for taking a look at this.

I'm using debian sarge packages on this system - the kernel patch
package for vserver says the version is 1.9.5.6. This host is using
the backports.org collectd packages (3.10.1-4).

vhost0:/proc/virtual# ls -la
total 0
dr-xr-xr-x 2 root root 0 2006-11-24 20:06 .
dr-xr-xr-x 289 root root 0 2006-11-21 06:47 ..
dr--r--r-- 2 root root 0 2006-11-24 20:06 49152
dr--r--r-- 2 root root 0 2006-11-24 20:06 49153
dr--r--r-- 2 root root 0 2006-11-24 20:06 49155
dr--r--r-- 2 root root 0 2006-11-24 20:06 49159
dr--r--r-- 2 root root 0 2006-11-24 20:06 49161
-r--r--r-- 1 root root 0 2006-11-25 04:40 info
vhost0:/proc/virtual# cd 49155
vhost0:/proc/virtual/49155# ls -la
total 0
dr--r--r-- 2 root root 0 2006-11-24 20:06 .
dr-xr-xr-x 2 root root 0 2006-11-24 20:06 ..
-r--r--r-- 1 root root 0 2006-11-24 20:06 cacct
-r--r--r-- 1 root root 0 2006-11-24 20:06 cvirt
-r--r--r-- 1 root root 0 2006-11-25 20:48 info
-r--r--r-- 1 root root 0 2006-11-24 20:06 limit
-r--r--r-- 1 root root 0 2006-11-25 20:48 sched
-r--r--r-- 1 root root 0 2006-11-25 20:48 status
vhost0:/proc/virtual/49155# cat cacct
UNSPEC: 0/0 0/0 0/0
UNIX: 19561/296937 19471/296937 0/0
INET: 14906/1453922 5044/1351263 1/9456
INET6: 0/0 0/0 0/0
OTHER: 32/12424 16/320 0/0
forks: 0
vhost0:/proc/virtual/49155# cat cvirt
BiasUptime: 4912.57
SysName: Linux
NodeName: www-dev
Release: 2.6.8+vserver
Version: #1 SMP Mon Nov 20 23:14:45 EST 2006
Machine: i686
DomainName: (none)
nr_threads: 98
nr_running: -126
nr_unintr: 0
nr_onhold: 0
load_updates: 33359908
loadavg: 980693.06 1303.90 89952.13
total_forks: 4374
vhost0:/proc/virtual/49155# cat limit
PROC: 98 112 -1 0
VM: 238927 273423 -1 0
VML: 0 0 -1 0
RSS: 34665 58507 -1 0
ANON: 0 0 -1 0
FILES: 434 568 -1 0
OFD: 234 345 -1 0
LOCKS: -944 -1 -1 0
SOCK: 87 107 -1 0
MSGQ: 0 0 -1 0
SHM: 181 181 -1 0

Thanks again!

-Kevin

[Sebastian Harl]

Hi Kevin,

Post by kcheek at umich.edu ()
I'm using debian sarge packages on this system - the kernel patch
package for vserver says the version is 1.9.5.6.

Uh! This is a _really_ old release. Please consider upgrading to a current
version (which should be available from backports.org). Nevertheless, your
files actually look fine (besides some really strange details - see below) and
I currently don't have any clue what your problem might be caused by.

Please make sure that the vserver plugin is enabled in the config file.

If I find some spare time tomorrow, I will build a debugging version of the
vserver plugin...

Post by kcheek at umich.edu ()
nr_running: -126

This tells you, that you have -126 threads running in your Linux-VServer
guest. Thus you should have at least -126 CPU's in your computer ;-)

Post by kcheek at umich.edu ()
loadavg: 980693.06 1303.90 89952.13

That is a _really_ huge load average... never seen anything similar before ;-)

[...]

I guess these oddities are caused by your outdated Linux-VServer version.

Cheers,
Sebastian
--
Sebastian "tokkee" Harl
GnuPG-ID: 0x8501C7FC
http://tokkee.org/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
Url : http://mailman.verplant.org/pipermail/collectd/attachments/20061126/9ba09823/attachment.pgp

[kcheek<span style="display: none;">.trailing-username</span>(a)<span style="display: none;">leading-domain.</span>umich.edu ()]

Post by Sebastian Harl
Hi Kevin,

Post by kcheek at umich.edu ()
I'm using debian sarge packages on this system - the kernel patch
package for vserver says the version is 1.9.5.6.

Uh! This is a _really_ old release. Please consider upgrading to a current
version (which should be available from backports.org). Nevertheless, your
files actually look fine (besides some really strange details - see below) and
I currently don't have any clue what your problem might be caused by.
Please make sure that the vserver plugin is enabled in the config file.

I just discovered that the vserver rrd's were created when using
"Local" mode, but they weren't created when using client/server mode.

It hadn't occurred to me before that enabling plugins on the server
side would be relevant, but enabling the plugin on the server did
appear to fix the problem. I had assumed that enabling plugins was
only relevant on the client since the server wasn't "using" the
plugin, just collecting the data it received from the clients.

-Kevin

[unknown]

Hi,

I had assumed that enabling plugins was only relevant on the client since
the server wasn't "using" the plugin, just collecting the data it received
from the clients.

Well, the server _is_ using the plugin in any case as only the plugin knows
how to process the data and write it to the rrd files. Maybe this should be
mentioned explicitly in the manpage?

Cheers,
Sebastian
--
Sebastian "tokkee" Harl
GnuPG-ID: 0x8501C7FC
http://tokkee.org/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
Url : http://mailman.verplant.org/pipermail/collectd/attachments/20061128/7bf0c12e/attachment.pgp

[Florian Forster]

Hi Kevin,

Post by kcheek at umich.edu ()
collectd[9116]: apache: curl_easy_perform failed: Failed writing body
[...]
127.0.0.1 - - [25/Nov/2006:15:15:44 -0500] "GET /server-status?auto HTTP/1.1" 200 4289 "-" "collectd/3.10.3" "-"

that's your problem right there: Apache returns 4289 bytes, but collectd
uses only a 4096 byte buffer internally. This isse has come up once
before and the next minor version will use a 16k byte buffer, hoping
that noone has *this* many processes ;)

Post by kcheek at umich.edu ()
Scoreboard: __W__.................. [...]

This is the cause for this problem: Many many many process slots ;) You
basically have four choices:
1) Wait for the next version, which is due sometime in december, the
exact date depends on my workload.
2) Use the development version.
3) Re-compile collectd with a bigger buffer.
4) Decrease the number of process slots by about 200. Unless your
webserver is _very_ busy this shouldn't be a problem.

Post by kcheek at umich.edu ()
So far, I'm also unable to get statistics with the vserver or mysql
plugins, but I'm not seeing any error messages in syslog for those -
they just don't produce any rrd files.
The mysql.log shows messages like "061125 20:32:10 7 Query SHOW GLOBAL
STATUS" every ten seconds, so I'm guessing that collectd is able to
talk to mysqld.

Sounds like it _should_ work.. The `mysql' should at least create the
traffic graph or report an error. I hardly have the heart to ask this,
but: You are aware, that the mysql-RRD-files are in a seperate
subdirectory? Other than that I don't know where it should go wrong :/

Post by kcheek at umich.edu ()
I'm not sure where to look for problems with the vserver plugin.

Sebastian is by far the more competent contact for this ;)

Regards,
-octo
--
Florian octo Forster
Hacker in training
GnuPG: 0x91523C3D
http://verplant.org/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
Url : http://mailman.verplant.org/pipermail/collectd/attachments/20061126/3e6f76f6/attachment.pgp

[kcheek<span style="display: none;">.trailing-username</span>(a)<span style="display: none;">leading-domain.</span>umich.edu ()]

Post by Sebastian Harl
Hi Kevin,

Post by kcheek at umich.edu ()
Scoreboard: __W__.................. [...]

This is the cause for this problem: Many many many process slots ;)
[...]
4) Decrease the number of process slots by about 200. Unless your
webserver is _very_ busy this shouldn't be a problem.

Do you know where that is configured? If it's an apache compile-time
option, then I guess I'll either wait for the next version or try the
development version. The apache configuration settings I tried didn't
seem to have an impact on the number of slots in the scoreboard
(MaxClients, MaxRequestsPerChild, MaxSpareServers, MinSpareServers,
StartServers, and maybe some others).

-Kevin

[unknown]

Hi Kevin,

Post by kcheek<span style="display: none;">.trailing-username</span>(a)<span style="display: none;">leading-domain.</span>umich.edu ()

Post by Florian Forster
4) Decrease the number of process slots by about 200. Unless your
webserver is _very_ busy this shouldn't be a problem.

Do you know where that is configured? If it's an apache compile-time
option, then I guess I'll either wait for the next version or try the
development version.

no, it's not a compile time setting. If I recall correctly, the setting
to control the number of process-slots is called `ServerLimit'. For
details read here:
<http://httpd.apache.org/docs/2.0/mod/mpm_common.html#serverlimit>

Hope this helps. A short feedback, if this in fact was the correct
option would be apprechiated.

Regards,
-octo
--
Florian octo Forster
Hacker in training
GnuPG: 0x91523C3D
http://verplant.org/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
Url : http://mailman.verplant.org/pipermail/collectd/attachments/20061128/4e2b974b/attachment.pgp

[kcheek<span style="display: none;">.trailing-username</span>(a)<span style="display: none;">leading-domain.</span>umich.edu ()]

Post by unknown

Post by kcheek<span style="display: none;">.trailing-username</span>(a)<span style="display: none;">leading-domain.</span>umich.edu ()

Post by Florian Forster
4) Decrease the number of process slots by about 200. Unless your
webserver is _very_ busy this shouldn't be a problem.

Do you know where that is configured? If it's an apache compile-time
option, then I guess I'll either wait for the next version or try the
development version.

no, it's not a compile time setting. If I recall correctly, the setting
to control the number of process-slots is called `ServerLimit'. For
<http://httpd.apache.org/docs/2.0/mod/mpm_common.html#serverlimit>

It looks like that option isn't available pre-2.0 (still running 1.3
on most systems here). Yes, we're not exactly on the bleeding edge
around here for many things.

I tried moving to apache2 on a test system. Without changing any of
the default settings (other than enabling mod_status), collectd seems
perfectly happy with the output. I guess I can add another item to my
todo list.

-kevin