CRE8LEARN INSTITUTE

C6-LIE-19 Linux Essentials
Section C Techniques Reference

Administration, automation and resilience (15 marks)

Instructor: Teboho Mpholo (Mr.), Founder and Lead Instructor
CP3 is Monday 2 November: Section C started

How to use this page. These are techniques with made-up names (a group called vets, a project in ~/demo). Your role model, your table, your unit files and your explanations must come from your own machine and your own design. Near-identical work between students is referred for collusion. The ACL, mask, setgid, umask, sudoers, tar and purge examples were tested on GNU/Linux. The systemd unit files could not be run where this page was prepared, so check them with systemd-analyze --user verify on your machine.

C1. Permissions, ACLs and least privilege 6 marks

1. Find out what really exists on your machine

getent passwd | awk -F: '$3>=1000 && $3<60000 {print $1, $3, $7}'   # human accounts
getent group | awk -F: '{print $1, $3}' | sort -k2 -n | head -40
id; groups; id -nG

On a lab machine you usually see only yourself plus system accounts (nobody, daemon and so on). Say which real accounts or groups stand in for clerks, officers and the administrator in your demonstration. A group you belong to (from id -nG) is the easiest stand-in for a role. The administrator role is you, as owner of the files.

2. Design the three-role table first

Build a table in your report: rows are files (data/*.dat, logs/audit.log, etc/naledi.conf, bin/, lib/), columns are clerk, officer and administrator, cells use r, a, e, d, x, and a final column says why. Two facts shape the design:

A file has one owner, one group and "other". Three roles with different rights on the same file cannot be expressed with those three slots. That is where mode bits run out and an ACL (a named entry per role) becomes the right instrument.
POSIX has no "append only" permission. The w bit allows append, edit and truncate alike (chattr +a needs root). Say so, then explain how your design reduces the risk: clerks get no write bit on the files and reach them only through the scripts, every change is audited, and backups exist.

3. Implementing roles with ACLs

cd ~/demo
setfacl -m g:vets:r-x bin lib                    # a group may run scripts but not edit them
setfacl -m g:vets:r-- data/loans.dat             # read only on one file
setfacl -d -m g:vets:r-- data                    # default ACL: files created later inherit it
getfacl -p data/loans.dat
setfacl -x g:vets data/loans.dat                 # remove one entry
setfacl -b data/loans.dat                        # remove all ACL entries

A + at the end of the ls -l mode string means the file carries an ACL. To remove a role's rights, do it with one command per file so your screenshots show a clear before and after.

4. Prove enforcement against yourself, and explain the mask

You cannot log in as someone else, so test on yourself. Tested behaviour matters here, because it surprises people:

touch data.dat
setfacl -m u:$USER:r-- data.dat ; echo x >> data.dat   # STILL works: the owner entry matches first
setfacl -m u::r-- data.dat      ; echo x >> data.dat   # Permission denied
getfacl -p data.dat                                    # shows user::r--
setfacl -m u::rw- data.dat      ; echo x >> data.dat   # restored, write succeeds

Why: for the file's owner Linux uses the owner entry user::, never a named entry for the same account. The mask does not limit the owner entry either, so to deny yourself you change user::. To show the mask doing its job, use a named entry for another account or group:

touch f2 && chmod 664 f2
setfacl -m u:nobody:rw f2
ls -l f2        # -rw-rw-r--+ 
setfacl -m m::r-- f2                                   # tighten the mask only
ls -l f2        # -rw-r--r--+  (the group bits now show the MASK)
getfacl -p f2   # user:nobody:rw-   #effective:r--

This is the lesson for the report: ls -l shows the mask where you expect the group bits, so it hides that nobody was granted rw- and that only r-- is effective. Only getfacl shows the entry, the mask and the #effective result together. Screenshot each state with the banner.

5. setgid, sticky bit and umask: create a file after each change

mkdir sg; chgrp vets sg; chmod 2770 sg      # drwxrws---   (the s in the group slot)
touch sg/new plain
stat -c '%A %U:%G %n' sg sg/new plain
# tested: sg/new gets group vets (inherited), plain keeps your own primary group

(umask 027; touch u1; mkdir u2; stat -c '%A %n' u1 u2)   # -rw-r-----  drwxr-x---
mkdir st; chmod 1770 st; stat -c '%A %n' st              # drwxrwx--T  (T: sticky, no other-execute)

setgid on a directory: new files inherit the directory's group, so every role's files stay in the group that the ACLs are written for. It prevents files falling into someone's private group and becoming unreadable to the role that needs them.
Sticky bit: in a directory many users may write to, only a file's owner (or the directory owner) can delete or rename it. It belongs on a shared drop directory (like /tmp), not on data/, where nobody but the administrator should be deleting at all. Decide where it fits your deployment and say why.
umask 027: new files get no access for "other" at all. Put the line in ~/.bashrc and show the effect on a new file.

6. The sudoers fragment you would install (not installed)

# /etc/sudoers.d/naledi-officers   (edit only with: visudo -f /etc/sudoers.d/naledi-officers)
Cmnd_Alias NALEDI_RUN = /opt/naledi/bin/naledi --run-compliance [0-9][0-9][0-9][0-9]-[0-9][0-9]
Cmnd_Alias NALEDI_LOG = /usr/bin/tail -n 200 /var/log/naledi/audit.log
%officers ALL = (naledi) NOPASSWD: NALEDI_RUN, NOPASSWD: NOEXEC: NALEDI_LOG

This one parses cleanly with visudo -cf. Justify every line in your own words. The ideas to cover:

Absolute paths: sudo compares the full path. A bare name would be looked up through the caller's PATH, so a user could place their own program first and run it with the extra rights.
Fixed arguments, not *: a wildcard also matches spaces, so * would let an officer append other options. The glob above allows only a YYYY-MM month.
The script must not be writable by officers. If they can edit the allowed program, they can put anything in it. Root-owned, mode 755 or stricter.
Reading logs: a fixed tail on one file is safer than journalctl or less, which can open a pager and then a shell. NOEXEC blocks the command from launching other programs.
Run as naledi, not root, so a mistake cannot reach the rest of the system. No ALL anywhere. No shell, editor or interpreter in the list.

sudo visudo -cf /etc/sudoers.d/naledi-officers     # syntax check before it goes anywhere
sudo -l -U someofficer                              # lists exactly what that user may run
sudo -l                                             # then test an allowed and a forbidden command

Permissions on the file itself should be root:root and mode 0440. A syntax error in a live sudoers file can lock everyone out of sudo, which is why you validate first and keep a root session open.

C2. Automation and scheduling 4 marks

1. A user timer (check it works on your machine first)

systemctl --user is-system-running; ps -p 1 -o comm=; echo $XDG_RUNTIME_DIR

If PID 1 is systemd and systemctl --user answers, use a user timer. If it says it cannot connect to the bus, say so and use crontab instead.

# ~/.config/systemd/user/naledi-sweep.service
[Unit]
Description=NALEDI permit expiry sweep
[Service]
Type=oneshot
ExecStart=%h/demo/bin/naledi --sweep-permits

# ~/.config/systemd/user/naledi-sweep.timer
[Unit]
Description=Daily NALEDI sweep
[Timer]
OnCalendar=*-*-* 02:30:00
Persistent=true
[Install]
WantedBy=timers.target

systemd-analyze --user verify ~/.config/systemd/user/naledi-sweep.service ~/.config/systemd/user/naledi-sweep.timer
systemctl --user daemon-reload
systemctl --user enable --now naledi-sweep.timer
systemctl --user start naledi-sweep.service         # run it once now
journalctl --user -u naledi-sweep.service --no-pager | tail

%h is your home directory in a unit file, so no ~ or shell is needed. A literal % must be written %%.
The time must come from naledi.conf (TIMER_TIME). Systemd cannot read it, so write a small script that generates the timer file from the config value, and say so in the report.
For the monthly run, let your entry point treat a missing month as the current month. That keeps the unit free of shell tricks and still calls the same entry point as the dashboard.
Output goes to the journal by default. A cron job needs >> ~/demo/logs/cron.log 2>&1 and your own rotation.

2. Prove it works with a stripped environment

env -i /bin/bash -c '~/demo/bin/naledi --sweep-permits'     # no HOME, no USER, no profile
env -i PATH=/usr/bin:/bin /bin/bash ~/demo/bin/naledi --sweep-permits   # cron-like

Tested: under env -i the variables HOME and USER are unset, and bash supplies only a default PATH. Your job must still work, which is why the scripts find their own root and the config holds absolute paths (A3). Cron runs with an even smaller PATH than your terminal.

3. Scheduling state and the boot chain

systemctl --user list-timers --all
systemctl --user cat naledi-sweep.service naledi-sweep.timer
systemctl is-system-running
systemd-analyze blame | head -15
crontab -l                                  # if you used cron

4. Missed-run recovery

Set Persistent=true, schedule the timer for a time when the machine is off or you are logged out, then in your next session show the catch-up. Systemd records the last trigger in a stamp file, which is useful evidence:

ls -l --time-style=full-iso ~/.local/share/systemd/timers/
journalctl --user -u naledi-sweep.service --since yesterday --no-pager
systemctl --user list-timers --all          # check the LAST column
uptime -s; date

Logged out and not running? A user timer only runs while your user manager runs. loginctl enable-linger $USER keeps it alive without a login, but it may be refused on a shared lab machine, and if so say so and use catch-up logic instead. A tested catch-up pattern for cron:

catch_up() {   # run from cron every 30 minutes, safe to run many times
  local stamp="$ROOT/logs/.last_run" today; today=$(date +%F)
  [ "$(cat "$stamp" 2>/dev/null)" = "$today" ] && return 0       # already done today
  [ "$(date +%H%M)" \< "0230" ] && return 0                      # not yet time
  run_job && echo "$today" > "$stamp"                           # stamp only after success
}

Persistence proof (log out and in, or reboot): one screenshot showing uptime -s, date, list-timers and the job's recent log lines together.

5. Timer or cron, for your report

Think about a shared machine that is switched off overnight. For each of the two, write one operational advantage and one disadvantage in your own words. Points to consider: what happens to a job whose time passed while the machine was off, where logs and failure status go, what runs without a login session, how dependencies and the environment are declared, and how much you can inspect afterwards.

C3. Backup, retention and a proven restore 5 marks

1. Archive and manifest

backup_now() {
  local d a; d=$(date +%F); a="naledi_$d.tar.gz"
  tar czf "$ROOT/backup/$a" -C "$ROOT" data etc logs
  ( cd "$ROOT/backup" && sha256sum "$a" > "$a.sha256" )
}
( cd "$ROOT/backup" && sha256sum -c naledi_2026-11-02.tar.gz.sha256 )     # verify

Schedule it with the same timer pattern as C2. Also keep a manifest of the files inside (find data -type f -exec sha256sum {} + | sort -k2 > data.manifest), because the archive checksum only proves the archive, not the restored files.

2. Retention with "first of each month kept"

purge_backups() {
  local dir="$ROOT/backup" f ym seen=" " keep="$ROOT/.keep.$$"
  : > "$keep"
  for f in $(ls "$dir"/naledi_*.tar.gz | sort); do          # oldest first
    ym=$(basename "$f" | cut -d_ -f2 | cut -c1-7)
    case "$seen" in *" $ym "*) ;; *) seen="$seen$ym "; echo "$f" >> "$keep";; esac
  done
  find "$dir" -name 'naledi_*.tar.gz' -mtime +"$RETENTION_DAYS" | sort | while read -r f; do
    if grep -qxF "$f" "$keep"; then echo "KEEP (first of month) $(basename "$f")"
    else echo "PURGE $(basename "$f")"; rm -f -- "$f" "$f.sha256"; fi
  done
  rm -f "$keep"
}
# make test data with deliberate ages
for d in 2026-07-03 2026-07-20 2026-08-02; do : > backup/naledi_$d.tar.gz; touch -d "$d 02:30" backup/naledi_$d.tar.gz; done

Tested with seven archives dated July to October 2026: the first archive of July, August and September survived, the later July and August ones were purged, and everything inside the 30-day window stayed. RETENTION_DAYS must come from naledi.conf. Show a dry run first (print only), then the real purge.

3. The destructive restore drill

cd ~/demo; wc -l data/*.dat | tee ~/demo/evidence/counts.before
T0=$(date +%s); date -Is; ls -l --time-style=full-iso backup/ | tail -3     # last backup time
rm -rf data;       date -Is; ls data                                         # deleted, error shown
tar xzf backup/naledi_2026-11-02.tar.gz -C ~/demo data
( cd backup && sha256sum -c naledi_2026-11-02.tar.gz.sha256 )
wc -l data/*.dat | diff - evidence/counts.before && echo "COUNTS MATCH"
bin/naledi --sweep-permits; T1=$(date +%s); echo "RTO seconds: $((T1-T0))"; date -Is

RPO is the time between your last good backup and the moment of the disaster (the data written in between is what you would lose), so read both from your own timestamps. RTO is the time from the deletion to a verified, working system, which is the number above. Quote your measured figures, then say what would change them (a more frequent timer, a faster disk, less data).

4. A filesystem that does not keep ownership

tar tvzf backup/naledi_2026-11-02.tar.gz | head -5   # owner, group and mode are stored inside
cp backup/naledi_2026-11-02.tar.gz /media/$USER/MYSTICK/
cp -a ~/demo/data /media/$USER/MYSTICK/data_plain    # compare the warnings and the result
stat -c '%A %U:%G %n' ~/demo/data/*.dat /media/$USER/MYSTICK/data_plain/*.dat
cp /media/$USER/MYSTICK/naledi_2026-11-02.tar.gz /tmp/ && mkdir /tmp/r && tar xzf /tmp/naledi_2026-11-02.tar.gz -C /tmp/r
stat -c '%A %U:%G %n' /tmp/r/data/*.dat

Why plain copy fails: FAT, exFAT and NTFS do not store Linux owners or mode bits. The mount invents one owner and one mode for every file, and cp -a can only copy what the target can store, usually with "failed to preserve" warnings.
Why tar survives: the archive is one ordinary file, and the owner, group and mode of each member are written inside it. The foreign filesystem never has to understand them. Copy the archive back to a Linux filesystem and extract there.
Ownership on extraction: tested as root, a normal extract restores the stored owner (here tester:vets), and --no-same-owner gives every file to the extracting user (here root:root). As an ordinary user, files are always yours, so say what you saw and what --no-same-owner would change if an administrator did the restore. -p keeps the stored mode bits ignoring your umask.

What the marker will test

Item	Likely checks
C1 [6]	Accounts and groups actually found on your machine. Table of roles with reasons. Mode bits insufficient where, and why ACL. Refusal shown with `getfacl` and the effective mask, then restored. Each of setgid, sticky and umask shown by a new file. Every sudoers line justified, validated with visudo and sudo -l.
C2 [4]	Unit files or crontab in full. Same entry point as the dashboard. Works under `env -i`. list-timers, cat, is-system-running and blame shown. A missed run recovered with timestamps. Persistence across sessions with uptime and date in one screenshot. Timer versus cron comparison.
C3 [5]	Dated archive and manifest. Purge keeps the first of each month. Deletion proven before restore. Manifest verified, counts matching. The cross-filesystem journey explained and shown. RPO and RTO measured, not guessed.