6.2 Uploading and security

Overall idea

To reduce the CPU burden on the shared host (as well as some security concerns), we do not compile all of LilyPond. The website build process runs texi2html, but all media files (be they graphical lilypond output, photos of people, or pdfs) are copied from the $LILYPOND_WEB_MEDIA_GIT repository.

All scripts and makefiles used for the website build are run from a “trusted” copy. Any modification to those files in git needs a human to review the changes (after they have been made in git) before they are used on the server.

Building the website (quick local)

Initial setup: make sure that you have the environment variables $LILYPOND_GIT and $LILYPOND_WEB_MEDIA_GIT set up correctly. For more information, see Other repositories.

Once that is done,

cd $LILYPOND_GIT/build/
make website

The website is in ‘out-website/website/index.html’.

Building the website (exactly as on the server)

Setting up (exactly as on the server)

Initial setup: you still need $LILYPOND_GIT and $LILYPOND_WEB_MEDIA_GIT.

Once that’s done, create:

mkdir -p $HOME/lilypond/
mkdir -p $HOME/lilypond/bin/
mkdir -p $HOME/lilypond/cron/
mkdir -p $HOME/lilypond/trusted-scripts/

The add these files to ‘$HOME/lilypond/bin/’:

Update git repositories:

### update-git.sh
#!/bin/sh
cd $LILYPOND_GIT
git fetch origin
git merge origin/master
cd $LILYPOND_WEB_MEDIA_GIT
git fetch origin
git merge origin/master

Check for any updates to trusted scripts / files:

### check-git.sh
#!/bin/sh
DEST=$HOME/lilypond/trusted-scripts
diff -u $DEST/website.make \
  $LILYPOND_GIT/make/website.make
diff -u $DEST/lilypond-texi2html.init \
  $LILYPOND_GIT/Documentation/lilypond-texi2html.init
diff -u $DEST/extract_texi_filenames.py \
  $LILYPOND_GIT/scripts/build/extract_texi_filenames.py
diff -u $DEST/create-version-itexi.py \
  $LILYPOND_GIT/scripts/build/create-version-itexi.py
diff -u $DEST/create-weblinks-itexi.py \
  $LILYPOND_GIT/scripts/build/create-weblinks-itexi.py
diff -u $DEST/mass-link.py \
  $LILYPOND_GIT/scripts/build/mass-link.py
diff -u $DEST/website_post.py \
  $LILYPOND_GIT/scripts/build/website_post.py
diff -u $DEST/bib2texi.py \
  $LILYPOND_GIT/scripts/build/bib2texi.py
diff -u $DEST/langdefs.py \
  $LILYPOND_GIT/python/langdefs.py
diff -u $DEST/lilypond.org.htaccess \
  $LILYPOND_GIT/Documentation/web/server/lilypond.org.htaccess
diff -u $DEST/website-dir.htaccess \
  $LILYPOND_GIT/Documentation/web/server/website-dir.htaccess

If the changes look ok, make them trusted:

### copy-from-git.sh
#!/bin/sh
DEST=$HOME/lilypond/trusted-scripts
cp $LILYPOND_GIT/make/website.make \
  $DEST/website.make
cp $LILYPOND_GIT/Documentation/lilypond-texi2html.init \
  $DEST/lilypond-texi2html.init
cp $LILYPOND_GIT/scripts/build/extract_texi_filenames.py \
  $DEST/extract_texi_filenames.py
cp $LILYPOND_GIT/scripts/build/create-version-itexi.py \
  $DEST/create-version-itexi.py
cp $LILYPOND_GIT/scripts/build/create-weblinks-itexi.py \
  $DEST/create-weblinks-itexi.py
cp $LILYPOND_GIT/scripts/build/mass-link.py \
  $DEST/mass-link.py
cp $LILYPOND_GIT/scripts/build/website_post.py \
  $DEST/website_post.py
cp $LILYPOND_GIT/scripts/build/bib2texi.py \
  $DEST/bib2texi.py
cp $LILYPOND_GIT/python/langdefs.py \
  $DEST/langdefs.py
cp $LILYPOND_GIT/Documentation/web/server/lilypond.org.htaccess \
  $DEST/lilypond.org.htaccess
cp $LILYPOND_GIT/Documentation/web/server/website-dir.htaccess \
  $DEST/website-dir.htaccess

Build the website:

### make-website.sh
#!/bin/sh
DEST=$HOME/web/
BUILD=$HOME/lilypond/build-website
mkdir -p $BUILD
cd $BUILD
cp $HOME/lilypond/trusted-scripts/website.make .

make -f website.make WEBSITE_ONLY_BUILD=1 website
rsync -raO $BUILD/out-website/website/ $DEST/website/
cp $BUILD/out-website/pictures $DEST
cp $BUILD/out-website/.htaccess $DEST

Then in the ‘cronjob/’ directory, put the cronjob to automate the trusted portions:

# website-rebuild.cron
LILYPOND_GIT=   ... fill this in
LILYPOND_WEB_MEDIA_GIT=   ... fill this in

11 * * * * $HOME/lilypond/trusted-scripts/update-git.sh >/dev/null 2>&1
22 * * * * $HOME/lilypond/trusted-scripts/make-website.sh >/dev/null 2>&1

As the final stage of the setup, run your copy-from-git.sh script, assuming that you trust the current state of scripts in lilypond git.

Normal maintenance

When there is a change to the build scripts and/or website makefile, log in to the server (or your own home machine if you’re testing this there), and do

update-git.sh
check-git.sh

After reviewing the changes carefully, you can update the trusted scripts with copy-from-git.sh.

Building the website (exactly as on the server)

Run make-website.sh; the final version ends up in ‘$HOME/web/’.

On the actual server, the website is generated hourly by user graham the host lilypond.org. You can set up the cronjob by doing:

crontab $HOME/lilypond/website-rebuild.cron

Initial setup for new users on actual serve

You should symlink your own ‘~/lilypond/’ to ‘~graham/lilypond/’

If this directory does not exist, make it. Git master should go in ‘~/lilypond/lilypond-git/’ but make sure you enable:

git config core.filemode false

If you have created any files in ‘~graham/lilypond/’ then please run:

chgrp lilypond ~graham/lilypond/ -R
chmod 775 ~graham/lilypond/ -R

Additional information

Some information about the website is stored in ‘~graham/lilypond/*.txt’; this information should not be shared with people without trusted access to the server.

<< Back to Documentation Index

LilyPond — Contributor’s Guide