Collectd Graph Panel v0.3

117 Replies

It has been a while and many people were already using one of the development versions of CGP. Time to release a new version of CGP: v0.3.

Special thanks for this version go to Manuel CISSÉ, Jakob Haufe, Edmondo Tommasina, Michael Stapelberg, Julien Rottenberg and Tom Gallacher for sending me patches to improve CGP. Also thanks to everyone that replied after the previous release.

In this version there are a couple of minor improvements and some new settings to configure. But the most important: Support has been added for 13 Collectd plugins. CGP now supports 26 Collectd plugins. Also a couple of plugins have been updated. Please read the changelog or git log for more information about the changes.

New plugins:

Download the .tgz package, the patch file to upgrade from v0.2 or checkout the latest version from the git repository.

Download: http://pommi.nethuis.nl/…/cgp-0.3.tgz
Patch: http://pommi.nethuis.nl/…/cgp-0.2-0.3.patch.gz
Git: http://git.nethuis.nl/pub/cgp.git

117 thoughts on “Collectd Graph Panel v0.3”

  2. poisonbit

    This is one of the best frontend I’ve found, that does not require CGI, ruby, python, or other requeriments…

    Untar and use in any LAMP server… nice job.

    mysql plugin could be nice 🙂

  3. pommi Post author

    @xiz: I have added support for the cpufreq plugin, check the development version. 🙂 For the battery plugin I have to search for a machine with a battery. It will be added soon.

    @poisonbit: I tried to enable/configure the collectd mysql plugin, but I hit a problem:
    lt_dlopen (/usr/lib/collectd/mysql.so) failed: file not found

    I don’t know what is wrong, because mysql.so is present at that location. I’m running Collectd 4.10.1(-1+squeeze1~bpo50+1) on Debian Lenny. I hope to fix this soon so I can add support for the MySQL plugin too.

  5. pommi Post author

    After upgrading MySQL to the version from Debian backports I was able to use the Collectd MySQL plugin. Probably Collectd from backports was build against MySQL from backports. 😕

    Support for the Battery and MySQL plugin have been added to the development version.

    If you notice any problems, please report them over here or send me a patch to fix the problem.

  6. poisonbit

    MySQL plugin Tested here and working perfectly.

    I fetched ‘http://git.nethuis.nl/?p=cgp.git;a=snapshot;h=refs/heads/master;sf=tgz’ and tested it andI see all the mysql graphs.

    A lot of thanks. I will recommend this frontend to every friend that needs one.

    🙂

  10. innate_ideas

    Great frontend to collectd! Thanks for the work you put into it.

    A minor correction is needed in plugin/cpu.php. rrd_vertical is set to ‘Jiffies’ but should be ‘Percent’.

  12. Weboide

    This is a VERY useful tool! I like how it’s working directly out of the box.
    Have you ever thought of creating an entry on freshmeat.net, sourceforge, or ohloh?

  13. xiz

    Hi Pommi,

    Mysql, battery and cpufreq work fine with the version I just grabbed from GIT.
    Thanks again!

  17. pommi Post author

    Poil, thanks a lot for the idea of expanding/collapsing/refreshing all graphs. Also thanks for including a patch! When I have time, I will review it and possibly include it in the next release.

  19. Remi

    Has anyone had a chance to test out the apache plugin on apache2.0?

    In testing in our environment, everything works fine except the graph description. For all graphs it simply states “Scoreboard for on HOST”. Changing the following line in apache.php makes it currently return “Scoreboard for apache_scoreboard on HOST”

    Original:
    $obj->rrd_title = sprintf(‘Scoreboard of %s’, $obj->args[‘pinstance’]);

    New:
    $obj->rrd_title = sprintf(‘Scoreboard of %s’, $obj->args[‘type’]);

  21. Remi

    I just went to the latest and things look much better thanks!

    I also merged in Poll’s changes, very welcome additions!

  23. Björn Abheiden

    CGP gives a great overview of collectd data.
    Do you have a developers guide to write own plugins? I want to plot some graphs from our software. How can I use our data stores with your tool?

  24. pommi Post author

    Hi Björn, CGP only creates graphs from the Collectd data directory (the RRD files). If your “software” is a Collectd plugin, CGP should be able to create graphs when you write a CGP plugin for it. Just copy-paste and modify an another CGP plugin to start with.

  25. Björn Abheiden

    Hi pommi,

    thank you … but …
    My “software” writes files into the rrd directory of collectd but it is no collectd plugin.

  26. pommi Post author

    Hi Björn, does your software write/update RRD files in the rrd directory? The use of RRD files is a prerequisite. If you use the same structure as Collectd, you should at least be able to view the plugin name in CGP. And when you expand your plugin in CGP you should see error images. If you get this far, you can start writing a CGP plugin in the plugin directory of CGP. Just copy pieces of code from existing CGP plugins to get yours working.

  27. Björn Abheiden

    Yes, the software writes rrd files into the rrd directory, but the new directory is not listed as a plugin. CGP asks collectd on installed plugins and get a list of them. In my opinion this is the reason why our new directoy will not shown.

  28. daedalus

    First, thanks for a very good piece of software. Secondly a request: Any chance of a DNS plugin?

  29. Xan

    Error:No Collectd hosts found in /var/lib/collectd/rrd

    All seems to be correct: the server, and collectd, but CGP show me that error. How can I fix it?

    Thanks,
    Xan,

  30. pommi Post author

    Hi Xan, This message can mean 2 things:
    1. Your collectd datadir is not in /var/lib/collectd/rrd. Set $CONFIG[‘datadir’] in conf/config.local.php to set the correct one.
    2. Your webserver user (for example: user www-data when Debian/Apache2) is not allowed to read the files in /var/lib/collectd/rrd

  32. Xan

    This is the 2: permissions. Perhaps you could put a debug code in CGP for displaying correctly if it’s 1 or 2. Thanks,
    Xan,

  33. Xan

    nginx plugin does not show in CGP (in dev version)

    Hi,

    I downloaded via git the last revision of CGP. And it does not show nothing…. what happens?

    Regards,
    Xan.

  35. Tilo

    Hi pommi,

    first i have to say thank you for that great tool, for me it’s the best visualization of collectd’s data.

    On my 64bit lenny box all things run out of the box, but for my 64bit squeeze I only get a segmentation fault. I don’t know very much about running php scripts at console but it is the same error as in the apache log.

    [1] 4444 segmentation fault (core dumped) php5-cgi index.php

    I will send you the created dump, if you let me know where to send. I’ve also tried to clone your git tree, but git ends with that message.

    fatal: http://git.nethuis.nl/pub/cpg/info/refs not found: did you run git update-server-info on the server?

    Sometimes it is weird.

    Thank you in advance,
    Tilo

  37. Tilo

    Thank you for your fast reply! The segfault came from bare copy of config.php to config.local.php – ouch! It is a kind of stupid what php’s include handling does. They should work on it 🙂

    The second was the typo. Now everything works fine with the current dev-branch! Thanks!

  40. Tilo

    Hi pommi,

    what Jab~ probably means is a link to your site. Something like that in ./inc/html.inc.php:39

    Collectd Graph Panel is distributed under the GNU General Public License (GPL)

    I think this would be a great benefit 😉 I’ve also search several times and this would be much faster.

    My actual question:
    Are you planning to develop a libvirt plugin? In my case this would be a greater benefit. Neither I’m having a good understanding of rrdtool nor a good starting point to overcome. If you could give me one I will see what i can do …

    Regards,
    Tilo

  42. pommi Post author

    Hi Poil, please don’t post plugin code in the comments. You can send it by e-mail to me or publish it somewhere and post the link in the comments.

    But thanks anyway for the code. I do not fully understand it, because you are talking about a PostgreSQL plugin, but the code is about the filecount plugin. Also there is a require_once of modules/collectd.inc.php, but this file does not exist.

    Based on your code I will try to implement the filecount plugin asap.

    Stacking multiple CPU’s in 1 graph is also on my wishlist. 🙂 What makes it more difficult is that it will not be a basic implementation. All plugins are implemented really straight-forward. Maybe when more plugins need multiple instances grouped in 1 graph it is interesting to implement this feature.

  43. Poil

    Thanks for your reply, I haven’t change header of my file, it’s postgresql plugin.
    I’ll mail you filecount plugin which I’ve also made.

    I’m using your graph class in a homemade software (source code will be release, when it’ll be working but it’s not all people ready,), so some path will not be OK. You can remove collectd.inc.php.

    I’ve wrote netapp plugin but this one need to Stack some line and to draw other not stacked. I think today there is no way to do this easily.

    Best Regards

  45. pommi Post author

    Hi Poil, thanks a lot for the plugins. I started integrating them in CGP, starting with the PostgreSQL plugin. Update your git repository or download the latest development version to check it out. 😉

  47. yur

    first of all, thank you for the very nice (the best) rrd frontend on PHP.
    the only feature I was missing is total calculations in IO graphs, so I have modified the GenericIO type like the following.

  48. pommi Post author

    Hi yur, thanks for your addition. I’ve included the patch. (But please don’t post code in the comments.)

    @Tilo: I didn’t forget about the libvirt plugin. It will be added soon.
    @Poil: The PostgreSQL and Filecount plugins are implemented 🙂 Can you please provide a 100% CGP compatible NetApp plugin? Unfortunately I don’t have NetApp stuff to test the plugin, but the plugin you provided doesn’t work correctly.

  49. Andy

    Great tool. Would you be able to include the tail plugin? This is my primary focus for collectd and cgp.

  50. Andy

    I copied another plugin and experimented until I got it to work, but I could use some pointers on setting the scale appropriately.

    What is rrd_format %.1lf?

  51. pommi Post author

    Hi Andy, I’ll add support for tail soon. The rrd_format variable is used to format the values in the legend. More information: rrdgraph_graph.

  52. beorn

    Hi pommi,

    I did write a patch that might interest you. How do i send it to you ?

    beorn

  53. pommi Post author

    Hi Beorn, just put it somewhere online and leave a link in the comments. Or send it by email (my address is just like the URL of this site, replace the 1st dot with an @.

  61. pommi Post author

    Michael, thanks for the Exec plugin example. I’ve not been able to run Collectd 5 yet, but CGP should have support for Collectd 5 already. Have you tried to set $CONFIG['version'] to 5 in your conf/config.local.php?

  62. michael

    Hi pommi 🙂

    have set the version to 5, but the graph for df is not there.
    I have a vserver with collectd4 and compared the df directory to my local collectd5.

    my collectd5 has it like this df-root/df_complex-free.rrd (3files free,used,reserved)
    and collectd4 is just df/df-root.rrd with only one file. Guess they changed that plugin alot.

  63. pommi Post author

    Finally I got Collectd 5 working on my Linux Debian machine. 🙂 I’ve done some changes to make CGP compatible with Collectd 5. Please download the latest development version and set $CONFIG['version'] to 5.

  64. michael

    well done! df displays correctly now 🙂
    now i can look at more pretty graphs hehe

  66. Tilo

    @pommi Thank you, it works!
    I had a busy time in last weeks, so excuse the delayed reply. 🙂

  67. Steve Dibb

    Thanks for the awesome program, works great. 🙂

    I did notice, that if PHP error_reporting with Notices is turned on, it will not generate some graphs because of the notices … some variables not being set, for example. The graphs will just show up as broken images.

    Once I disabled error_reporting, and stopped displaying_errors they worked fine.

    So, any PHP errors in the code will cause a graph to not display. I can send a small patch of a few vars I’ve fixed if you like.

  68. pommi Post author

    @Steve: Thanks for your comment. I didn’t notice this problem yet, but I’m not running all plugins. It would be great if you provide a patch. Please send it to pommi.nethuis.nl (replace 1st dot with an @).

  69. Jan

    Hi Pommi,

    I already use Jarmon and works for me, but I love to try CGP.

    However, with both the latest development version and 0.3, it doesn’t show any graphs on my system (Ubuntu 10.04LTS) using firefox. With IE and Chrome it shows a broken image. The error.log from apache doesn’t give a clue either.

    As I only recently installed php (from source) for using wordpress, I wondered if i maybe missed some dependencies. Or if you have some other suggestions about what might be wrong.

    Thanks in advance.

  71. Jan

    Hi Pommi,

    Yes, I have collectd (incl. rrdtool) running. I already use Jarmon to display the data.

    The strangest thing is that when testing (different apache configs, CGP settings etc.) once in a while I do see graphs (twice till now). However i cannot reproduce it; more like luck that I see a graph. But when afterwards Jarmon is checked I immediatly loose the graphs with CGP again.

    Might there be some kind of conflict, where Jarmon and CGP are looking to the same RRD files?

  73. pommi Post author

    Jan, sorry for my delayed reply. There couldn’t be a conflict between Jarmon and CGP as far as I know. They are just reading the RRD datafiles. I also haven’t seen this problem before. Did you try other browsers? Are you able to show a single graph (the PNG image) and refresh it multiple times? Is the image always shown?

  74. Poil

    Hi,
    In the libivrt collectd plugin If you use
    HostnameFormat "hostname name"

    You’ll need to patch Generic***.class.php to escape colonns and add simple quote

    "'".str_replace(':','\:',$this->files[$tinstance])."'"

  76. Noah

    Nice work. With collectd v5, you will need to update plugin/apache.php

    index 95c12fa..b08b885 100644
    — a/plugin/apache.php
    +++ b/plugin/apache.php
    @@ -153,6 +153,7 @@ $obj->width = $width;
    $obj->heigth = $heigth;
    $obj->rrd_format = ‘%5.1lf’;

    +/*
    # backwards compatibility
    if ($CONFIG[‘version’] data_sources = array(‘count’);
    @@ -165,6 +166,7 @@ if ($CONFIG[‘version’] colors[‘value’]);
    }
    }
    +*/

    collectd_flush($obj->identifiers);
    $obj->rrd_graph();

    Or manually set $CONFIG[‘version’] in conf/config.php to 5.

    Failure to do so results in:

    ERROR: No DS called ‘count’ in ‘/var/lib/collectd/localhost/apache-local/apache_bytes.rrd’
    ERROR: No DS called ‘count’ in ‘/var/lib/collectd/localhost/apache-local/apache_requests.rrd’
    ERROR: No DS called ‘count’ in ‘/var/lib/collectd/localhost/apache-local/apache_bytes.rrd’
    ERROR: No DS called ‘count’ in ‘/var/lib/collectd/localhost/apache-local/apache_idle_workers.rrd’
    ERROR: No DS called ‘count’ in ‘/var/lib/collectd/localhost/apache-local/apache_connections.rrd’
    ERROR: No DS called ‘count’ in ‘/var/lib/collectd/localhost/apache-local/apache_scoreboard-open.rrd’

    -Noah

  79. pommi Post author

    Hi Poil, that looks like a nice piece of software 🙂 It unfortunately doesn’t work out-of-the-box for me. Is there some kind of INSTALL/README file with short instructions how to install it?
    Btw. why didn’t you fork my git repo instead of creating a new git repo with a big “initial” commit. It would have been easier to port stuff between CGP and CGRAPHZ.

  81. pommi Post author

    Thanks for the installation guide. About git, I would probably do something like this:

    Clone http://git.nethuis.nl/pub/cgp.git
    Remove all CGP files (don’t remove .git directory)
    Copy all CGRAPHZ files in the root
    Create your first CGRAPHZ commit

    Now you keep the CGP history and can do a “git show” to see the differences between CGP and CGRAPHZ.

  82. Poil

    Ok, after 3 tries, here it is !
    https://github.com/Poil/CGraphz

    1- Clone : git clone http://git.nethuis.nl/pub/cgp.git cgraphz
    2- cd cgraphz/
    3- git rm -r ajax.js host.php detail.php plugin.php index.php layout inc doc conf .htaccess
    4- git config –global user.name “Poil”
    5- git config –global user.email Poil@EMAIL.COM
    6- git remote add cgraphz git@github.com:Poil/CGraphz.git
    7- git push -u cgraphz master
    8- git commit -a # Commit my RM
    9- git push -u cgraphz master
    10- cp -r ../Poil/CGRAPHZ/cgraphz/[a-z]* . # Copy cgraphz file over cgp
    11- git add config/ html/ img/ index.php lib/ modules/ plugin/curl.php plugin/getjmx.php plugin/httpreponsetime.php plugin/icinga.php plugin/jmx_values.php plugin/netapp.php plugin/oracle.php plugin/pnp_ping.php sql/ style/ #Add all cgraphz files and folder
    12- git commit -a
    13- git push -u cgraphz master

    I hope that I’ve forgot nothing

  84. Bill

    Trying to get HEAD cup running on FreeBSD 8, with Apache22+extensions
    and Collectd 5.. Got everything working to the point of showing my hosts, and then it’s giving no graphs, just the broken img icon.. Any suggestions where to look? Works great on Ubuntu..

  85. pommi Post author

    Hi Bill, first make sure you have set $CONFIG[‘version’] in conf/config.local.php. Check your Apache error.log when something looks wrong.

  87. Philipp

    I can not get the nginx plugin running… even with the latest git clone version I get:

    FastCGI sent in stderr: “CGP Error: plugin “nginx” is not available”

  90. pommi Post author

    Beorn, jsrrdgraph looks like a great library! Nice features, like zooming and going back and forward in time. jsrrdgraph renders graphs client-site, which is a lot faster (in my case) than rrdgraph server-site. It also means the rrd files need to be available to the browser. A little disadvantage is there will be an increase of traffic (by default 150~300K per rrd file). Thanks for the comment, Beorn. This library will definitely be integrated in CGP.

  91. warp.kawada

    hello

    I submit my patch.
    And there is not libvirt.php in 0.3 tarball.
    Please append it.

  92. pommi Post author

    Hi warp.kawada, please don’t post big chunks of patches/code in the comments. Now I have the patch, can you please explain what your patch fixes? Download the latest development version to use the libvirt plugin.

  93. Poil

    Hi,

    How do you used socket/rrdcached ?

    I have run a tcpdump, I’ve got this request :
    FLUSH plugin=rrdtool identifier=”systest4t/load/load”

    But if I played it manually (telnet myrrdcached) I’ve :
    -1 No such file: /var/lib/collectd/rrd2/plugin=rrdtool.

    If I run this it’s working : FLUSHsystest4t/load/load.rrd

    Best regards

  95. Poil

    Hi Pommi,

    Oh OK … I was thinking that socksplugin is capable to flush rrdcached.
    I’ve rewrite your flush fonction to flush remote rrdcached socket, it’s a little dirty, I will commit it on cgraphz.

    Perhaps we can add $CONFIG[‘PROTO’] native_collectd or rrdcached
    Collectd4/5 + native = your flush function
    Collectd4/5 + rrdcached = my dirty one
    Collectd5.1 = your (or new one?)

  96. BillLortz

    I tried to use this package with OpenWrt and their Luci_statistics package and wasn’t able to get any graphs. It took me a while, but I found two small issues that prevented the graphs. I’m posting this to save some time for others.

    OpenWrt by default only creates AVERAGE RRDs to save memory. This prevents graphs from being produced when CGP requests MIN and MAX values. Fortunately, the box I was running OpenWrt on had plenty of memory and I unchecked the box labeled “Only create average RRAs”. That fixed most of the graphs.

    It seems like a patch could be developed that inspects the RRD using RRDTOOL’s INFO command and only tries to show data for what is in the RRD. When I have more time, I might try to produce one.

    I still had a problem with interface statistics not showing any graphs. It turns out the version of RRDTOOL included in OpenWrt doesn’t support the VDEF graph command. I commented out the lines of code in your tool that related to “VDEF” and magically the graphs appeared.

    I am using OpenWrt’s Backfire edition.

    Thanks for the great tool.

  97. K

    Just like Xan above I get:

    Error: No Collectd hosts found in /var/lib/collectd/rrd

    I checked the path and there’s information stored, I can even retreive it using KCollected. The path is correct and set to full permissions just for testing, even changed group to www-data, but the error persists.

    I get this from doing a tail /var/log/apache2/error.log:
    PHP Warning: is_dir(): open_basedir restriction in effect. File(/var/lib/collectd/rrd) is not within the allowed path(s): (/var/www/vhosts/collectd/collectd.local:/usr/share/php:/usr/share/pear) in /var/www/vhosts/collectd/collectd.local/httpdocs/inc/collectd.inc.php on line 11, referer: http://collectd.local/

    Tried to find a way around it looking at .htaccess and php.ini but found no working solution. Not much fun so far. Any ideas?

    Thanks

  99. Tilo

    Hi pommi,

    I need a Plugin for powerdns, is it possible for you to manage it?

    Please let me know how I can give you back a little bit (Flattr, Amazon Wishlist, Paypal or whatever) My company runs your Frontend since May 2011 and I’m pretty sure they are willing to honor your work. It saved us so much time.

    Regards,
    Tilo

  100. pommi Post author

    Tilo! Thanks for your comment. Support for the powerdns plugin will be there as soon as I have some time. There is a donate button on the right now. Feel free to donate something 😉

  102. Poil

    Hi Pommi,

    It’s nice to see you again 🙂
    I’ve add some plugins on Cgraphz, feel free to use them. (GenericJMX, Mongo …)
    https://github.com/Poil/CGraphz/tree/master/plugin

    Oh and I have a bug on GenericJMX, I think it’s in my code (not CGP), RRD’s filenames of generic JMX have whitespace and CGraphz truncate the p/pi/t/ti parameters on these. I have to replace whitespace by underscore in a collectd postrules.

    Best regards

  103. pommi Post author

    Tilo! I’ve added a very basic PowerDNS plugin today. Download the latest development version over here.

  105. Estêvão Samuel Procópio Amaral

    Hi, all,

    I started to use CGP today and I’m really liking it! One thing I missed was the possibility to navigate bak in time on the graphs. I did it and didn’t find where to send the patch. Are you using any bug tracker?

    Thanks!

  107. xian310

    Hello,

    First of all, thank you very much for your excellent and very useful tool CGP.

    For my personal use, I have made some changes in the navigation of CGP by creating a plugins menu. In my opinion, this allows a better navigation on the application.

    You can find 2 screenshoots of the changes here:

    http://www.gutsofdarkness.com/perso/xian/cgp/cgp1.png
    http://www.gutsofdarkness.com/perso/xian/cgp/cgp2.png

    A patch containing the changes is available at:

    http://www.gutsofdarkness.com/perso/xian/cgp/cgp.patch

    Let me now if you are interested in incorporate this changes in the official version of CPG.

  109. Reza

    Hi!

    First, nice job. I have installed your frontend and I really like it 😉

    Second I faced a little problem which seems to be a little bug.

    Of cource, my collectd folder is outside of my web root. I have added my collecd Folder to the open_base directive, so far so good.

    Your script “collectd.inc.php” calls the “is_dir” function before checking if it is the parent folder link, that means that your script works but also throws a Warning that the parent folder of the collectd root folder is outside the allowed folders.

    So I have changed the boolean condition in the script “collectd.inc.php” on line 16 from

    if( !is_dir($CONFIG[‘datadir’].’/’.$v) || $v == ‘.’ || $v == ‘..’ )

    to

    if( $v == ‘.’ || $v == ‘..’ || !is_dir($CONFIG[‘datadir’].’/’.$v))

    You should update your revision to get rid of that problem. 😀

    best regards, Reza

  110. xian310

    Hello,

    Here is a patch for a very simple implementation of a page auto-refresh mecanism.

    A configuration directive ($CONFIG[‘auto-refresh’]) allows setting the refresh interval in seconds.

    The patch is available at:

    http://www.gutsofdarkness.com/perso/xian/cgp/cgp2.patch

    Let me now if you are interested to incorporate this feature in the official version of CGP.

  111. martos

    Hi,

    Is there a way to turn graphics for “interface” in bites per second?

    Thanks in advance!

  114. Karel

    Hi,
    i have testing CGP on openwrt, but i have this error. Where is wrong? 🙂
    Parse error: syntax error, unexpected ‘typesdb’ (T_STRING) in /mnt/SD/monitoring/CGP-master/conf/config.php on line 10

  115. pommi Post author

    Hi Karel, what is on line 10 in config.php? This?
    $CONFIG['typesdb'][] = '/usr/share/collectd/types.db';
    Can you try to change it to:
    $CONFIG['typesdb'] = array('/usr/share/collectd/types.db');

  116. bobby

    Hello,

    I installed CGP everything works fine when i want to see localhost graphs but not for remote hosts. On the index.php page * i can see the new VM server connected with the average load but when i click on the VM server (In this case its called CentOS 7 #1) all i see is a page with “Unknown host” written **, the page is called “host.php?h=CentOS+7+%231”. Is there a for the remote VM graphs to show up?

    I have pics so you know what i’m talking about, One* indicates pic #1 and two **
    indicates pic #2
    1. https://imgur.com/Lqyr8QP
    2. https://imgur.com/L7uHw5O

    +Firewall is down for client and master servers
    +All new VMs
    +CentOS 7 for client and Debian 7.5 for master

    Thanks

Leave a Reply

Your email address will not be published. Required fields are marked *