Well, I will have an interesting
So, lets begin with the agenda
I will talk a little bit about So, lets begin with the agenda
I will talk a little bit about
the history,
what makes Munin
very unique,
we'll see
unique but not in a very good way we'll see
what makes it also really unique but not in a very good way
what makes
unique in
2.0 version, what
we got in Wheezy,
it's very interesting.
we will also see
according to the new features
of 2.0 you can scale
much more a Munin
install
install from
1.4 package
and we will see also a limitation
of 2.0 since
now you can scale quite well
I mean,
theorically, you can scale really well,
practically, well,
we will see. We still have some
big big issues
very different from
the ones in 1.4, but
still, and I will
present rapidly
the roadmap for
2.2.
Hopefully, it's released
this year. Well,
that's a challenge.
Thing is, if you have
any questions, I will
stop at 10:00, and
we will have 15 minutes for
questions just after.
So, brief history,
Munin was born in 2002, and was named RRD
I didn't know that fact before
and I just know it because I researched
for the presentation
and it's not a well known fact but
some code
I mean, most code
still dates from
that day. So it's quite
important to see that
issues
when changing code
it's more like geology.
You have every
every layers
If you want to add a functionality
one layer newer functionality
one layer... Well,
you all know this. [laughter]
So, I hacked zooming
for 1.2 in 2007
I mean
to 1.2
was very static
And well,
I maintained it in my
own private
place
And in 2009
1.4
came out, and I asked
if I could send my
patch to Munin
and, well, they got
accepted, and from
2009 until
2011, so, I was
slowly
gaining ground in the
Munin community until now
where, well,
I've just
took over the leadership
from the previous team
and it didn't
happen officially, but just
it's just the way it is.
so in 2012 I released
2.0
thanks to Holger
who said, "hey,
you have to release now, otherwise
you will release in ten years".
So, thanks to him, things were
very very
hectic at the early days
of 2.0
because I realized that
the biggest
point was
since it wasn't released
we didn't have many testers
and since we
didn't have many testers, I didn't want to
release it, since I still have some
bugs that
came out and so on.
So, thanks to Holger we broke
this cycle.
And we released
in 2012
it's interesting since it's
ten years since
it's born
and someone said that
every software gets good
after ten years, maybe,
and it's in
Wheezy since September
2012
and
it's in Stable
since Wheezy got out.
So, in
2013,
I released
2.1
it's unstable
a branch, because
I didn't want to have the
same problems as with
2.0, its
lack of testers, so I just packaged something,
I packaged the development branch
and released it. It's unstable.
Normally it works
but well, it's unstable.
You know what
unstable means.
And the biggest thing
is the Internet will
change in
...in the
2.1 lifeline
And I said, October
2013 is target for
release 2.2
But time will tell.
I mean, if you don't fix timelines and deadlines
you will never release things.
So, better be late than never.
Ok, so
a very simple
design principle of Munin is
I really love this quote from
Allan Kay, it's
"Simple things should be simple,
complex things should be possible."
That's exactly the motto of Munin.
Munin makes
simple things simple, and
complex things simple things simple, and
complex things
possible
It's very easy to use, it has
the same out-of-the-box
behavior, because when you
install it on a server, it
automatically starts monitoring.
And if not, it's a bug.
Please report it.
And, it
has a complete plug-and-play infrastructure
compared to
others, you...
The only thing you need
is to declare
the node, because, well,
broadcasting on
the local network
is not very practical
in my point of view, so it's the only thing
you need to say.
You have to poll this node, and
the node will just
hand all the config
to the master
so th graphs
are drawing.
Thing is,
our user
The vast majority of
our users, just has one server
to monitor, and
it's the same that
the Munin install is on it.
I mean, that's
that's why
the default Munin
is always targetted at this user.
But if you have a bigger install
Well, you already know
how to change
config files, usually.
And, as I said,
some are running bigger
installs, and
this are
the ones that interest me
very much in 2.2, since
well, we do
address really well the
one-node install type
but
for bigger installs
we
we have very much problems.
We improved
very much from
1.4 to
2.0, but now we
hit other limits that we will discuss
just after.
Ok, so, new features:
We really have now full
CGI implementation!
I mean, the one in
1.5 you should not
use it, I mean, it works,
sometimes, and it's
bugged every time.
And, so
it has also a full FastCGI
implementation
This is very important to
have adequate performance
so you don't reload everything
The biggest selling point
is, it has complete integration
with RRDcacheD
We will talk more about this
later, but this
is *the*
main issue
when scaling.
Because our RRD is
very nice, but doesn't scale
very well in its native
without RRDcacheD.
Thing is, when you use
RRDcacheD with some guidelines
that I will describe
later, and
you should not do
what you...
It has native ssh transport
Well, it's
before you used
plain TCP for
the connection
port 4949
You could use TLS
But most people didn't.
And, with native ssh,
usually people also
already use ssh on their
installs, so setting
ssh transport for them is quite easy
whereas having a TLS
thing is, you have to have a
certificate and so on and so on, it's...
Quite much more complicated.
And it's another open new port, as I said,
and it's secure
it's usually more integrated
than
in setups.
The other big feature is
async proxy. It's something
that sits on
the node,
that holds the node autonomously,
and stores
locally on the node,
the
Munin update
will then connect to the
async client part
and just repllay the spool that he
spooled just
before. So it has very
interesting
features. If you have
some nodes that have
loose connection, for example, you
want to monitor a remote
location that
has sometimes
no network or whatever
since it's locally spooling
when you connect you will
recover everything
that was
collected meanwhile you didn't
connect.
So, those little white bars
you were accustomed to are
gone.
It also speeds up poll.
Even for a local network
since
it delegates
all the polling and the waiting
for plugins
to the async proxy
the data collection
the Munin update
goes *really* faster
It only replays
a logged text file
So, the thing is that,
when you have a big cluster
it sometimes makes sense to
use async, since
well
the fixed five minutes
for Munin updates is
still a hard one, and
you cannot go further.
And one
last known thing about async proxy
is, it can
poll at various
update rates.
If you have one plugin
that says, "I want to be polled
every one hour",
async will only poll it
at one hour,
and the most interesting part,
if you have a plugin that says
"I want to be polled every ten seconds" if you have a plugin that says
"I want to be polled every ten seconds"
it will poll it every ten seconds, and still
send every
five minutes all of the data back
to the Munin update.
So, you won't have real-time
information
but you have very precise
information.
So now, we go
to scalability
That's the
biggest focus
on 2.0
because the first one was to
the zooming part
and zooming just
showed
that, well,
you can have huge data files
since
it doesn't... It's not very useful
to zoom on one year
history, if you
don't keep the
finer granularity in RRD
one year back
So
that will be scaling
data at the end. Really, what
people want is adding more nodes
That's the most common
scaling issue
that we get.
Inside the node, you can
also have a
huge number of plugins
some have very very large installations,
specially when you start to
use SNMP
because SNMP is done
by one host
to monitor many many
remote
routers, or
SNMP agents
and the thing is, some
also have slow plugins.
We already
discussed about
Munin update should take less than
five minutes, otherwise, well,
bad things happen
that's one point
and that's a hard rule
If your Munin updates
takes more than five minutes
really bad things happen
mostly white bars.
And so, if you have
many many plugins, and
many many plugins take
quite a long time to
poll, since it's all synchronous,
the fact is,
well,
it can go, even if you
parallelize
very much, it sometimes
still goes
quite slowly, and
if you
multiply the number of plugins
with long response times,
many nodes,
you have, you usually
pass the five minutes bar.
and it's scaling
data usually what's
with the zooming part
usually many many people
ask for, "well,
I can zoom one year ago
but now all I have
is one bar per day", I mean,
I don't care about the average
for one day
So here is
you can natively have
much
more data inside
we will see more about it later.
So, scaling the master
To have
a big install
the first thing is
use FastCGI.
Default is cron-based, remember
default is for the typical
user that has only one node and
one server,
any one that has
more than, lets say, five
nodes should really go
the CGI road
and not really CGI, but FastCGI
Because
The cron road
is, you generate every
kind of graphic, and it's
just pointless. I mean,
it's very simple, but it's pointless.
As I said,
You have to use
RRDcacheD, because, the thing is
RRD
is very, very nice, it's a
very nice piece of software, but
it has only one main problem
it is, it's so efficient
that it
writes only
the very little
part of the file
and to the underlying
I/O subsystem
RRD updates, when you have a big one,
it feels just like
random I/O. And when I say
random I/O, it's real random I/O.
I mean,
almost cryptographically
secure.
When I ask about
some
storage vendor,
he says, "random I/O, we can do that!"
I plugged Munin
with a big install on it
and he said, "what's that?"
Yeah, random I/O
"Well, not that random usually!"
The people of
RRD are well well aware of it
and even designed
RRDcacheD, that is specially
designed to make this
random I/O buffered
and to make it like normal
random I/O
And it's called,
there is a slide
you can Google it,
"RRDcacheD:
to escape the I/O hell"
it's really well described, and
to understand
what's behind
RRDcacheD
And it even works
on SSD, because usually
random I/O
"OK, no problem, just use SSD"
fast storage vendors
said, "well, no problem, we just put an SSD"
The thing is,
after
in my test, after four hours
with, yes, big installs,
all of the SSDs
were just
offline, because
too many I/Os
Because it writes, writes, writes, writes a lot.
So, SSD is
interesting, but not
only for us.
Thing is,
RRDcacheD
has only one
very big drawback, which is, you should never
ever read
from the RRD file
specially in cron, because
if you read on demand, it's perfect
it's only flushed
the file you are reading
but if you read it
in cron, by default,
you will read
the whoooole install
and that's exactly the same as
not using RRDcacheD, so...
So use it!
Thing is, for Munin,
you need lots of RAM.
Because, as I said,
we have RRDcacheD
but the more RAM
you put at RRDcacheD,
the longer you can keep
the spool
and so, the thing is,
it can
it writes very less often
and that's
very interesting.
If you have lots of RAM, you can
multiply the number of workers.
It means, you
obviously, if you have...
Since Munin is very very
very much I/O bound
either, so for waiting for NAS
or for waiting for the I/O subsystem
If you have
many workers, usually it
it helps
a lot, because every worker is
single-threaded.
Thing is,
but, do never
ever swap.
That's obvious. The thing is,
Munin is designed to
use all the memory
of its workers. So if
you only swap a little thing,
then there is no, how to say,
there is no lost memory.
You cannot swap...
For people that know the "swappiness" setting
it means swapping
before trading
some application memory
to file
cache memory
that's not a good idea
The application memory
is useful
at one point.
[... oops... That's OK ...]
On the master, you have really
to watch out for
shared hardware.
Because Munin is
very nice
and it loves
to annihilate any hardware
you put on it, because
...well...
It has
It is designed to be
very very scalable, and you can
launch as many processes as you
want, we will see
some kinds,
some limitations just after, but
it's designed to be very scalable
but the [?] arrives
not in a very efficient manner
I mean, it's not very clever
it just uses and goes
on your system.
So, for the record,
I have the
storage vendor
but was
[?] with all
of the
application of the
of the thing, and when we
wrote to it, I mean
99%
of the I/O ops were
delivered to the Munin server, so
let's imagine what
stays for the others.
Not much.
So, we put it on dedicated hardware,
it goes slower, but
well, other applications are happier.
The thing I said before,
is, use the async proxy.
Even you don't have a special
need on it,
thing is, it will
enable a very fast collection
As all the
I/O time or the wait
time is absorbed directly by
the async daemon,
your Munin
update almost
doesn't wait at all.
It only connects, reads a file
on the server
and disconnects. So,
while a usually typical
polling time is about
ten to
fifteen seconds
with Munin async, typical
time is about one second
to mosly two seconds.
Depends on
You have a
10 factor, that's
very interesting when you want to scale.
Because it lowers the
numer of update workers needed,
I said, Munin uses
lots of RAM, OK, but
usually you don't want to use RAM
for Munin update, you
prefer to use RAM for
the restitution part,
for graphs, for
the XML, of which we will speak
about later, and
Munin update you just want
it to be very quick and
if you don't have...
if it is
not I/O-bound from the network anymore,
it's only CPU-bound, and you
don't want to have more than the
CPU on your hardware
since over this is useless anyway.
And the thing that
a side effect but is very nice
it's, *if* your
Munin updates is very slow,
it happens,
and we speak about the five
minutes hard limit,
all the async-enabled
nodes will
not have any data loss.
You will have delays in integrating
the
the data, but you won't have
these infamous
white bars that most of you
already experience,
as these ones.
That was for
the server. For the node,
as I said, you
have, some have
a huge number of plugins.
The biggest install I
saw is about
one thousand plugins.
Wow.
It's
very interesting also
as async, because
it has the fork option, and async
knows it will just
prior to
to async
Munin update was
doing it very sequentially
and 1000 plugins
well, to have it
in less than
five minutes, it has to be quite fast.
Since it's not the only load
that is polled,
in async, with
the fork option,
each plugin will be
asked in its
own process, so
if you have
long running plugins,
as just after,
you also can use
the fork option,
before the plugin can
usually they poll themselves
either in cron or they just read the status back
that was the official way of doing it
in 1.4
but since async does exactly
that, in 2.0
just use async, I mean, it's
standard, and
it just makes using your
whatever you
use.
That's for the node.
Usually, the
only problem that
the node has when you have many many plugins
is the starting of the node
is typically serialized.
That's
When you have 1000 plugins, it's
a big problem.
Ok
Now, we are scaling the data.
As I said before,
zooming
brought the need of having prcise
data very
far away in time
and
to keep more data in RRD
it's very very easy
in 2.0
you have a new option
it's graph_data_size
you already
had it in
1.4
but it was global
now it is per plugin
it's also global, but you can
precise it per plugin
and
actually, it's designed to be
per field, but it
doesn't work, it...
it's bugged, and
mostly works per plugin, that's where
it works well, but it
only works on RRD create, so
there is an external tool to
move it
rota tool, that is called
RRD copy, to move from
some data, from
a small RRD, to a bigger
RRD.
But that's not part of
core Munin.
And when you create it
Its graphing is handled automatically by RRD
And RRD, as I said, is very very very
efficient, but
beware. It can use very
much space. I mean, I had
one person
who wanted to have
a ten-seconds precision for two years
...Wow.
It's about 500 megabytes
per RRD
so, per line
in Munin
Big data.
You can also increase RRD precision
it's called supersampling,
that works
without munin-async
if you put
munin-async, it will do
the job for you.
I will just go a little faster
since my time is almost up.
Bigger thing is
If you modify the
RRD size
always have
the RRA increased
because when you want
to have the graphs
if you
take huge, for example
there is a setting that is "huge",
it's not very...
These settings aren't very nice, because
it only has the
maximum precision for
two years, but
it doesn't have any RRA, and the RRA
are a part of Munin's
ability to reply
very fast on a yearly graph
for example. So if you
it's pre-consolidation
for yearly values.
So, the ideal way
you know the size of the graph,
in your templates,
and if you have one RRA
per pixel in the
graph outputted, it goes
fastest, since it doesn't
even have to interpolate the data.
So... Now,
the limitations of 2.0
The CGI of HTML
is very, very
very ugly.
I don't know yf many many of you
tried with big installs
but the practical limit is about
between 150
and 200 nodes.
After that, it's
very, very slow.
And it's slow
on reload, because
the whole configuration
is stored in a big
storable file
that is
that is reloaded
and most of the time is
took by
storable.reload
so I cannot do much about it.
We'll see how
I plan to do it.
The UI itself does not
... It is not very scalable.
I mean,
you all know the default
UI, so now
you have your cluster, just imagine
one thousand nodes inside
it's, well,
it's a little bit flat, and
not very...
All the nodes
are essentially
on the overview and
it's very static, and it's not
what one
does expect in 2013.
Because we all have
this web app,
and this
phone shining with
very dynamic stuff
And ours is not very dynamic, I agree.
The same is if you
you know the comparison page?
I mean, if every node
of a group and every graph, just imagine
that on one thousand nodes
one thousand plugins
Your Firefox
won't have any memory anymore.
And the last thing is
it lacks proper ACL.
For a bigger install,
usually you want to delegate
monitoring to
subsystems, and you don't
want people to see everything, because
it will be overwhelmed by
the information, and
well, that's
that's a problem.
So, I'll just go
very fast, that's my last slide
So, for 2.2 we will be
integrating into
2.1 and
when it is stable it will become 2.2
it's moving from
the whole Storable thing to
SQL-based, and
the SQL-based will be
DBI-based, because we are still in Perl
and will by SQLite
by default, because
we really want the
nice out-of-the-box
install, remember our users
[many] of them
are the one-node type,
and if you want, you can
do PostgreSQL
and if you want, you can do
whatever DBI supports
it's just up to you.
It will enable dynamic HTML
because
well, we are
not in 2001 anymore
But that will require
a deep rewrite of the code
As I said before,
when you have many many accessors
to Storable inside the core
but
since it was a big Storable,
it was a native Perl data structure
so for
whatever reason
many code does not use
accessors, they use it
in a typical Perl way
and that makes it very difficult
to translate to SQL.
So that's a challenge.
And just to be completely crystal-clear,
the data will stay,
the data that is in RRD will stay
in RRD. I don't want
to put the timestamped
value inside SQL.
That's not the point.
We will have a complete
node push
feature
The node can push
on the master
in order
well, to have
to break this five minutes pause
standard, so you can put
whenever you want, at every second if you want
and
this will enable
very fine
precision
and my goal is to be as good as
collectd.
If...
And well, if you have
that little
blurb on the new
HTML5 UI
I speeded up the end, so you have
a little time for questions.
If you want.
[question1] With this...
Is it possible, with this new
architecture
to
(sorry, I just missed my question)
[presenter] The SQL one, you mean? Or
[question] No, I had just wrote it down...
So, I'm sorry for...
[laughter]
[presenter] The async? Or...
[question1] Yes. Do you fork
the plugins?
The architecture still
forks the plugins
every time, or
is it possible to run
the plugin and keep it all running
and just feedback some values
as you will
you mentioned collectd, which
builds on this architecture.
[presenter]: I designed the
...a new extension
a new verb for plugins
it's called "stream"
and this is, you just
launch the plugin, you ask
for a config
and then you ask for the stream, and
when the plugin quits
it means, just sends
periodically values
at the rate he wants to
so it's very
...it is designed to capture, for example
the output of
vmstat
you can do
cat vmstat | awk
and, well, that's your
plugin output. It will stay
in memory. And
the plugin will kill himself
when the configuration changes.
That's
the design. But the promise
I didn't put it in 2.2
because I won't have time to do it.
But that's the way
it is done. But basically
the
architecture of forking/exec
a plugin
is, or will be
at the core of Munin. It won't be
for example
a DO .eso [?] or
that you will chart in, or .pm
that you will charge in
Munin memoryspace
That's
That's not something I want to.
[questioner1]: This was the
thing which
I really liked
Munin, and used it
in 1.2 or whatever, but
it had scaling problems
with regards of
work. So that was one of the reasons
I have to
change to another system.
[questioner2]: Hi.
So, I was a happy Munin user
And then, suddenly I
well,
because of the scaling issues,
I moved to a
pnp4nagios
and that's one
question I want to ask, is about how...
Because, with all this data
that is a great thing in Munin
then you can do proxy
monitoring
that is, sending OLS if [?] monitoring
that is, sending OLS if [?]
Do you plan on
having better integration with
[?] systems
than you have currently?
[presenter]: Actually
the point of
Nagios, I mean, we
We have very much problem with
because of
nscachanger
its interface, lately,
The thing is, we have
something called
munin-limits, and
it sends
[?] and so on, but it does not
do it very well.
So,
the integration with other systems
such as Nagios
icinger or whatever
is very very high on my top list
because I don't want to reimplement Nagios.
I mean, it is
I want to focus
on data gathering and
data keeping, I mean
I'm more interested in
replacing something like php4nagios
than Nagios itself.
[question2]: Becuase the
munin-limits, for example
it only has
threshold, like, if
it is a set of values, then 1
whereas I'm also interested in
questions like, OK,
usually this filesystem
is growing at 1% rate every
day, and suddenly it grew out like
50%. I want a warning there.
That's, you know...
[presenter]: Exactly, that's something
that is even offered by
RRD right now
and I also
have it on my future
roadmap, but, well,
I'm taking the problem
for user-facing
right now, but
everyone is welcome to help.
[question2]: I'm looking forward to it.
[presenter]: Yes, OK, so
time is up, so I guess you have to ask your questions
after the talk, and
thank you
[audience]: Just, there is a
BoF session
this afternoon, if you have some questions,
or anything specific, just come, and
I'll be glad to answer.
[presenter]: Thanks
[applause]