Bash Scripting 20 – Write date and data a file

So following on from the previous scripts, it may be useful to record both the date and some data associated with that date to a file.

I have found out how to do this, but also tidied my script up, removed some unneeded code (quite a lot) and this should work.

echo "date" "data" # for reference, not written to file 
for ((i = 0 ; i < 10 ; i++)); do
    echo $(date), $((RANDOM % 20)) # echo to screen (stdout)
    echo $(date), $((RANDOM % 20)) > log3.csv
done

Produces

date data
Fri 18 Apr 12:48:16 BST 2025, 0
Fri 18 Apr 12:48:16 BST 2025, 15
Fri 18 Apr 12:48:16 BST 2025, 13
Fri 18 Apr 12:48:16 BST 2025, 6
Fri 18 Apr 12:48:16 BST 2025, 6
Fri 18 Apr 12:48:16 BST 2025, 13
Fri 18 Apr 12:48:16 BST 2025, 18
Fri 18 Apr 12:48:16 BST 2025, 5
Fri 18 Apr 12:48:16 BST 2025, 3
Fri 18 Apr 12:48:16 BST 2025, 14

Which is fine, but we can refine the date output so it is shorter with

date "+DATE: %D TIME: %T"

  date "+%D"   display date no label
  date "+%T"  display tine no label 

If we want this in a csv file it may help to add a comma

date "+DATE: %D, TIME: %T"

Remove the labels DATE: & TIME:

date "+ %D, %T"

Produces

date "+ %D, %T"
 04/20/25, 14:58:19

My final entry, in the script run by cron, at least as far as time and date is concerned

echo $(date "+ %D, %T"), $(ansiweather -l Paignton -a false) >> weather.csv

As this sends the output to weather.csv, If we remove the redirect it will go to stout (standard output) we can just run the following on the terminal 'as is'

echo $(date "+ %D, %T"), $(ansiweather -l Paignton -a false)

This you should end up with

04/29/25, 15:44:14, Weather in Paignton: 16 °C - UVI: 5.91 - Wind: 0.89 m/s NNE - Humidity: 75% - Pressure: 1024 hPa

I have just copied and pasted the line in to the terminal and removed the >> weather.csv as that appends the csv file with the data.

As we all have different use cases, this solution is fine for me, I can adjust to suit further. This is far more useful for a CSV file, especially if you want to load the data in to a spreadsheet.

See fourth link below for full details. * Sources of help

https://stackoverflow.com/questions/40175868/how-to-create-csv-file-using-shell-script
https://stackoverflow.com/questions/1194882/how-to-generate-random-number-in-bash
https://linuxhandbook.com/bash-loops/
https://stackoverflow.com/questions/25242626/bash-redirect-output-to-log-file-with-day-date-in-the-name
https://phoenixnap.com/kb/linux-date-command

Tags

#Bash,#Bashscripting,#BashScripting,#Files,#Write,#Data,#Time,#Data

---

Mastodon	ShellLabs	Join Mastodon

AI statement : Consent is NOT granted to use the content of this blog for the purposes of AI training or similar activity. Consent CANNOT be assumed, it has to be granted.

Bash scripting 18 – Using data files 4

So, carrying on from the previous article(s).

Using the guide at [2]. We can set a few labels up so the graph looks nicer.

Once in gnu plot you can issue commands, then run replot to redraw the graph.

gnuplot> plot 'log.csv'
gnuplot> plot 'log.csv' with lines
gnuplot> set title 'Example Plot'
gnuplot> set xlabel 'x axis'
gnuplot> set ylabel 'y axis' 
gnuplot> replot
gnuplot> set key top right
gnuplot> replot

By running replot we can see the results are what we want before carrying on.

All this is just a small sample of what can be done, I will explore more on this once I have some useful, rather than randomly generated data to plot or do things with.

References

1 gnuplot 2 gnuplot examples

Chat

I am on the Devon and Cornwall Linux user group mailing list and also their matrix channel as zleap, it is better to ask there, that way others can answer too.

Tags

#Bash,#Bashscripting,#BashScripting,#Files,#UsingDataFiles,#Data,#gnuplot,#graphs

---

Mastodon	ShellLabs	Join Mastodon

AI statement : Consent is NOT granted to use the content of this blog for the purposes of AI training or similar activity. Consent CANNOT be assumed, it has to be granted.

Bash scripting 17 – Using data files 3

So, carrying on from the previous article(s). I am going to be using gnuplot to display the data. I found a useful site, that has examples of how to use the software, from basic to more advanced examples.

Notes: As with bash gnu plot has command history, so you can use arrow keys to move up / down between previously entered commands.

Based on the information at [2] we can produce a very basic plot of what is in log.csv

Load GNUPlot

gnuplot

and enter

plot 'log.csv'

This example isn't all that good really, but it at least produces some output. I have modified one of the examples further down that page

plot 'log.csv' using 1:2 with lines, 'log.csv' using 1:2 with lines

So the following data set

0, 1
1, 10
2, 10
3, 7
4, 2
5, 1
6, 15
7, 0
8, 19
9, 3

Should produce something like

Adding extra columns

We probably need more than two columns. This is easily done by modifying the loop in the script gendata1.csv. It is a good idea to copy this to a new script so it keeps the original as is

cp gendata1.sh gendata2.sh

As gendata1.sh has write permissions, these will be preserved in the new copy of the file. We can now edit gendata2.sh

echo "number" "data" # for reference, not written to file 
for ((i = 0 ; i < 10 ; i++)); do
    echo $i, $((RANDOM % 20)), $((RANDOM % 20)) # echo to screen (stdout)
	echo $i, $((RANDOM % 20)), $((RANDOM % 20)) >> log.csv # write to file
    #write_csv $(($i, $RANDOM % 10))
done

So all we are doing here is adding in, note the comma

, $((RANDOM % 20))

As the only difference between the lines

echo $i, $((RANDOM % 20)), $((RANDOM % 20)) # echo to screen (stdout)
echo $i, $((RANDOM % 20)), $((RANDOM % 20)) >> log.csv # write to file

Is that the first line, writes to the screen, it may be better to test the script out, check it does what you want, then modify the line that writes to a file once you are happy.

References

1 gnuplot 2 gnuplot examples 3. GNU Plot and LaTeX, included for reference

Chat

I am on the Devon and Cornwall Linux user group mailing list and also their matrix channel as zleap, it is better to ask there, that way others can answer too.

Tags

#Bash,#Bashscripting,#Files,#UsingDataFiles,#Data,#gnuplot,#graphs,#BashScripting

---

Mastodon	ShellLabs	Join Mastodon

AI statement : Consent is NOT granted to use the content of this blog for the purposes of AI training or similar activity. Consent CANNOT be assumed, it has to be granted.

Bash scripting 16 – Using data files 2

So, carrying on from the previous article(s). I am going to be using gnuplot and awk, and see what I can do with the data generated by my script.

It would be useful to be able to remove content in the file we are using with awk, so new data is not added to existing data.

We can do this by running:-

echo " " > log.csv

This should ensure the file is empty

we can then rerun

./gencsv1.sh 

And check contents with

echo csv.log

So according to the article, we can list the data in the 2nd column

awk -F, '{ print $2}' log.csv

 2
 7
 14
 4
 5
 7
 17
 19
 0
 19

and on a similar note, to list what is in the first column

awk -F, '{ print $1}' log.csv

and we can calculate the sum using

awk -F, '{ sum +=$2} END { print sum }' log.csv
94

and the average

awk -F, '{ sum +=$2} END { print sum/NR }' log.csv
8.54545

Chat

I am on the Devon and Cornwall Linux user group mailing list and also their matrix channel as zleap, it is better to ask there, that way others can answer too.

Tags

#Bash,#Bashscripting,#Files,#UsingDataFiles,#Data,#BashScripting

---

Mastodon	ShellLabs	Join Mastodon

AI statement : Consent is NOT granted to use the content of this blog for the purposes of AI training or similar activity. Consent CANNOT be assumed, it has to be granted.

Bash scripting 15 – Using data files 1

So, carrying on from the previous article. I am going to be using gnuplot and awk, and see what I can do with the data generated by my script.

So the first task, if of course to install gnu plot

sudo apt install gnuplot

It is also a good idea at this stage to determine version numbers of the tools we are using

gnuplot -V
gnuplot 6.0 patchlevel 0

and

awk -V
GNU Awk 5.2.1, API 3.2, PMA Avon 8-g1, (GNU MPFR 4.2.1, GNU MP 6.3.0)
Copyright (C) 1989, 1991-2022 Free Software Foundation.

This program is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation; either version 3 of the License, or
(at your option) any later version.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public License
along with this program. If not, see http://www.gnu.org/licenses/.

Tags

#Bash,#Bashscripting,#Files,#UsingDataFiles,#Data

---

Mastodon	ShellLabs	Join Mastodon

AI statement : Consent is NOT granted to use the content of this blog for the purposes of AI training or similar activity. Consent CANNOT be assumed, it has to be granted.

Bash scripting 14 – Generating data files

Issue 249, May 2025 of Linux Magazine, p52-55, had an article on using awk, and gnu plot to manipulate data. In order to try and make use of this, and learn more, I am trying to figure out how to write a csv file.

Write 10 lines to the first column should be in sequence e.g. 1-10 and the 2nd column should be random numbers. This should be written to a file.

So after some digging I managed to come up with this, by modifying some existing solutions

write_csv(){
    echo \"$1\",\"$2\" > log.csv # write to file, use >> to append
}

#sources of help
echo "number" "data" # for reference, not written to file 
for ((i = 0 ; i < 10 ; i++)); do
    echo $i, $RANDOM # echo to screen (stdout)
	echo $i, $RANDOM >> log.csv # write to file
    #write_csv $(($i, $RANDOM % 10))
done

The screen output for me was

number data
0, 17109
1, 2872
2, 22572
3, 7228
4, 27923
5, 6335
6, 20004
7, 32343
8, 10005
9, 21905

While the file output (log.csv) was similar but without the headings, which I just put in for my own reference, the file does work as a csv as I can open in LibreOffice Calc.

0, 17109
1, 2872
2, 22572
3, 7228
4, 27923
5, 6335
6, 20004
7, 32343
8, 10005
9, 21905

As the numbers are quite big, a closer examination, promoted me to make the following modification

echo $i, $((RANDOM % 20)) # echo to screen (stdout)
echo $i, $((RANDOM % 20)) >> log.csv # write to file

This fixes the 2nd column to produce a number between 1 and 20.

number data
0, 17
1, 12
2, 12
3, 16
4, 2
5, 13
6, 10
7, 6
8, 16
9, 16
cat log.csv
0, 3
1, 17
2, 8
3, 16
4, 10
5, 18
6, 5
7, 14
8, 11
9, 11

The script was called gencsv1.sh, which as expected needed execute permissions

chmod +x gencsv.sh

References

In order to help me I used the following, so giving credit to these sources as would be expected.

• https://stackoverflow.com/questions/40175868/how-to-create-csv-file-using-shell-script
• https://stackoverflow.com/questions/1194882/how-to-generate-random-number-in-bash
• https://linuxhandbook.com/bash-loops/

Tags

#Bash,#Bashscripting,#Files,#Random,#Data,#CSV,

---

Mastodon	ShellLabs	Join Mastodon

AI statement : Consent is NOT granted to use the content of this blog for the purposes of AI training or similar activity. Consent CANNOT be assumed, it has to be granted.

Bash scripting 13 – Linux permission table

As previously mentioned, scripts need to be given permission to run. This table should help with the process of working out what permission to set on a file or script.

I think the usual default is 755 which gives the user read, write, execute. Both the group and other users of the system get read and write.

For the most part, you can set execute permission on a script with

chmod +x scriptname.sh

	READ			WRITE			EXECUTE
USER	GROUP	OTHER	USER	GROUP	OTHER	USER	GROUP	OTHER
400	40	4	200	20	2	100	10	1

The markdown code for the above is

|         | **READ** |         |         | **WRITE** |         |         | **EXECUTE** |         |
|:-------:|:--------:|:-------:|:-------:|:---------:|:-------:|:-------:|:-----------:|:-------:|
|  _USER_ |  _GROUP_ | _OTHER_ |  _USER_ |  _GROUP_  | _OTHER_ |  _USER_ |   _GROUP_   | _OTHER_ |
| **400** |  **40**  |  **4**  | **200** |   **20**  |  **2**  | **100** |    **10**   |  **1**  |

Which may be useful. I used Tables Generator to help create this.

Chat

I am on the Devon and Cornwall Linux user group mailing list and also their matrix channel as zleap, it is better to ask there, that way others can answer too.

Tags

#Bash,#Bashscripting,#Files,#Permissions.#chmod,#Read,#Write, #Execute

---

Mastodon	ShellLabs	Join Mastodon

AI statement : Consent is NOT granted to use the content of this blog for the purposes of AI training or similar activity. Consent CANNOT be assumed, it has to be granted.

Bash scripting 12 – Files and Grep

Rather than make a video for this, I decided to just make a blog post so that I could include downloadable or at least copy / pasteable components.

Grep stands for GNU Regular Expression Parser, In essence and among other things, it can read (or parse) a file and report on contents, or in the case of this, find a specific string of text.

lorem Ipsum, is standard in the printing industry as it is dummy text used to fill on a page. I have pasted below an example and it just happens to explain further.

Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book. It has survived not only five centuries, but also the leap into electronic typesetting, remaining essentially unchanged. It was popularised in the 1960s with the release of Letraset sheets containing Lorem Ipsum passages, and more recently with desktop publishing software like Aldus PageMaker including versions of Lorem Ipsum.~

If you copy and paste the above and save in a text file called lorem.txt we can do some neat stuff with grep and a few other commands. I am not an expert at this, so this is some of what I picked up while researching this post.

Firstly we can find a specific word (or string) in the text with

cat lorem.txt | grep the

we can also do

grep the lorem.txt

Both will search the file lorem.txt, display the file contents and highlight the word 'the' from the text.

This is great, so what else can we do

In a short file, the number of times a word may appear may be less than 5 or 10. So we could just count manually. As discussed in a previous video, the command wc or word count, does what it says and counts the number of words.

So I found the following

cat lorem.txt | grep -o the | wc -l

Which gives the output as 6 which is how many times the word 'the' appears in the text.

As with other commands, there is a man page so

man grep

and

man wc

Should provide useful information, you can also search for information with duckduckgo and there are numberous tutorials on line.

Hope this is useful

Chat

I am on the Devon and Cornwall Linux user group mailing list and also their matrix channel as zleap, it is better to ask there, that way others can answer too.

Tags

#Bash,#Bashscripting,#Files,#TextSearch,#StringSearch,#Grep,#wc,#WordCount

---

Mastodon	ShellLabs	Join Mastodon

AI statement : Consent is NOT granted to use the content of this blog for the purposes of AI training or similar activity. Consent CANNOT be assumed, it has to be granted.

Resize and rename multiple files

As I was renaming and resizing some files anyway. I decided to make this short video of the process. I have added to Debian category as I am using Debian 11.

Commands used:-

gm -mogrify -resize 640x480 *.JPG AND rename 's/P1/codeclub/'

As usual your mileage may vary, you need to check man pages for specific arguments for your needs.

Happy to try and help further via Mastodon

Fedi id : zleap@qoto.org

Tags

#Debian11,#BASH,#Shell,#Rename,#Resize,#Files,#Eucation,#GNULinux

---

Mastodon	ShellLabs	Join Mastodon

AI statement : Consent is NOT granted to use the content of this blog for the purposes of AI training or similar activity. Consent CANNOT be assumed, it has to be granted.