Bash scripting 18 – Using data files 4

So, carrying on from the previous article(s).

Using the guide at [2]. We can set a few labels up so the graph looks nicer.

Once in gnu plot you can issue commands, then run replot to redraw the graph.

gnuplot> plot 'log.csv'
gnuplot> plot 'log.csv' with lines
gnuplot> set title 'Example Plot'
gnuplot> set xlabel 'x axis'
gnuplot> set ylabel 'y axis' 
gnuplot> replot
gnuplot> set key top right
gnuplot> replot

By running replot we can see the results are what we want before carrying on.

All this is just a small sample of what can be done, I will explore more on this once I have some useful, rather than randomly generated data to plot or do things with.

References

1 gnuplot 2 gnuplot examples

Chat

I am on the Devon and Cornwall Linux user group mailing list and also their matrix channel as zleap, it is better to ask there, that way others can answer too.

Tags

#Bash,#Bashscripting,#Files,#UsingDataFiles,#Data,#gnuplot,#graphs

---

Mastodon	Peertube	Join Mastodon

AI statement : Consent is NOT granted to use the content of this blog for the purposes of AI training or similar activity. Consent CANNOT be assumed, it has to be granted.

Bash scripting 17 – Using data files 3

So, carrying on from the previous article(s). I am going to be using gnuplot to display the data. I found a useful site, that has examples of how to use the software, from basic to more advanced examples.

Notes: As with bash gnu plot has command history, so you can use arrow keys to move up / down between previously entered commands.

Based on the information at [2] we can produce a very basic plot of what is in log.csv

Load GNUPlot

gnuplot

and enter

plot 'log.csv'

This example isn't all that good really, but it at least produces some output. I have modified one of the examples further down that page

plot 'log.csv' using 1:2 with lines, 'log.csv' using 1:2 with lines

So the following data set

0, 1
1, 10
2, 10
3, 7
4, 2
5, 1
6, 15
7, 0
8, 19
9, 3

Should produce something like

Adding extra columns

We probably need more than two columns. This is easily done by modifying the loop in the script gendata1.csv. It is a good idea to copy this to a new script so it keeps the original as is

cp gendata1.sh gendata2.sh

As gendata1.sh has write permissions, these will be preserved in the new copy of the file. We can now edit gendata2.sh

echo "number" "data" # for reference, not written to file 
for ((i = 0 ; i < 10 ; i++)); do
    echo $i, $((RANDOM % 20)), $((RANDOM % 20)) # echo to screen (stdout)
	echo $i, $((RANDOM % 20)), $((RANDOM % 20)) >> log.csv # write to file
    #write_csv $(($i, $RANDOM % 10))
done

So all we are doing here is adding in, note the comma

, $((RANDOM % 20))

As the only difference between the lines

echo $i, $((RANDOM % 20)), $((RANDOM % 20)) # echo to screen (stdout)
echo $i, $((RANDOM % 20)), $((RANDOM % 20)) >> log.csv # write to file

Is that the first line, writes to the screen, it may be better to test the script out, check it does what you want, then modify the line that writes to a file once you are happy.

References

1 gnuplot 2 gnuplot examples 3. GNU Plot and LaTeX, included for reference

Chat

I am on the Devon and Cornwall Linux user group mailing list and also their matrix channel as zleap, it is better to ask there, that way others can answer too.

Tags

#Bash,#Bashscripting,#Files,#UsingDataFiles,#Data,#gnuplot,#graphs,#BashScripting

---

Mastodon	Peertube	Join Mastodon

AI statement : Consent is NOT granted to use the content of this blog for the purposes of AI training or similar activity. Consent CANNOT be assumed, it has to be granted.

Bash scripting 16 – Using data files 2

So, carrying on from the previous article(s). I am going to be using gnuplot and awk, and see what I can do with the data generated by my script.

It would be useful to be able to remove content in the file we are using with awk, so new data is not added to existing data.

We can do this by running:-

echo " " > log.csv

This should ensure the file is empty

we can then rerun

./gencsv1.sh 

And check contents with

echo csv.log

So according to the article, we can list the data in the 2nd column

awk -F, '{ print $2}' log.csv

 2
 7
 14
 4
 5
 7
 17
 19
 0
 19

and on a similar note, to list what is in the first column

awk -F, '{ print $1}' log.csv

and we can calculate the sum using

awk -F, '{ sum +=$2} END { print sum }' log.csv
94

and the average

awk -F, '{ sum +=$2} END { print sum/NR }' log.csv
8.54545

Chat

I am on the Devon and Cornwall Linux user group mailing list and also their matrix channel as zleap, it is better to ask there, that way others can answer too.

Tags

#Bash,#Bashscripting,#Files,#UsingDataFiles,#Data,#BashScripting

---

Mastodon	Peertube	Join Mastodon

AI statement : Consent is NOT granted to use the content of this blog for the purposes of AI training or similar activity. Consent CANNOT be assumed, it has to be granted.

Bash scripting 15 – Using data files 1

So, carrying on from the previous article. I am going to be using gnuplot and awk, and see what I can do with the data generated by my script.

So the first task, if of course to install gnu plot

sudo apt install gnuplot

It is also a good idea at this stage to determine version numbers of the tools we are using

gnuplot -V
gnuplot 6.0 patchlevel 0

and

awk -V
GNU Awk 5.2.1, API 3.2, PMA Avon 8-g1, (GNU MPFR 4.2.1, GNU MP 6.3.0)
Copyright (C) 1989, 1991-2022 Free Software Foundation.

This program is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation; either version 3 of the License, or
(at your option) any later version.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public License
along with this program. If not, see http://www.gnu.org/licenses/.

Tags

#Bash,#Bashscripting,#Files,#UsingDataFiles,#Data

---

Mastodon	Peertube	Join Mastodon

AI statement : Consent is NOT granted to use the content of this blog for the purposes of AI training or similar activity. Consent CANNOT be assumed, it has to be granted.

Bash scripting 14 – Generating data files

Issue 249, May 2025 of Linux Magazine, p52-55, had an article on using awk, and gnu plot to manipulate data. In order to try and make use of this, and learn more, I am trying to figure out how to write a csv file.

Write 10 lines to the first column should be in sequence e.g. 1-10 and the 2nd column should be random numbers. This should be written to a file.

So after some digging I managed to come up with this, by modifying some existing solutions

write_csv(){
    echo \"$1\",\"$2\" > log.csv # write to file, use >> to append
}

#sources of help
echo "number" "data" # for reference, not written to file 
for ((i = 0 ; i < 10 ; i++)); do
    echo $i, $RANDOM # echo to screen (stdout)
	echo $i, $RANDOM >> log.csv # write to file
    #write_csv $(($i, $RANDOM % 10))
done

The screen output for me was

number data
0, 17109
1, 2872
2, 22572
3, 7228
4, 27923
5, 6335
6, 20004
7, 32343
8, 10005
9, 21905

While the file output (log.csv) was similar but without the headings, which I just put in for my own reference, the file does work as a csv as I can open in LibreOffice Calc.

0, 17109
1, 2872
2, 22572
3, 7228
4, 27923
5, 6335
6, 20004
7, 32343
8, 10005
9, 21905

As the numbers are quite big, a closer examination, promoted me to make the following modification

echo $i, $((RANDOM % 20)) # echo to screen (stdout)
echo $i, $((RANDOM % 20)) >> log.csv # write to file

This fixes the 2nd column to produce a number between 1 and 20.

number data
0, 17
1, 12
2, 12
3, 16
4, 2
5, 13
6, 10
7, 6
8, 16
9, 16
cat log.csv
0, 3
1, 17
2, 8
3, 16
4, 10
5, 18
6, 5
7, 14
8, 11
9, 11

The script was called gencsv1.sh, which as expected needed execute permissions

chmod +x gencsv.sh

References

In order to help me I used the following, so giving credit to these sources as would be expected.

• https://stackoverflow.com/questions/40175868/how-to-create-csv-file-using-shell-script
• https://stackoverflow.com/questions/1194882/how-to-generate-random-number-in-bash
• https://linuxhandbook.com/bash-loops/

Tags

#Bash,#Bashscripting,#Files,#Random,#Data,#CSV,

---

Mastodon	Peertube	Join Mastodon

AI statement : Consent is NOT granted to use the content of this blog for the purposes of AI training or similar activity. Consent CANNOT be assumed, it has to be granted.

US Cloud Services and EU citizens data

In a major move in the right direction for data protection and user privacy.

It is no longer safe to move our governments and societies to US clouds

The very short version: it is madness to continue transferring the running of European societies and governments to American clouds. Not only is it a terrible idea given the kind of things the “King of America” keeps saying, the legal sophistry used to justify such transfers, like the nonsense letter the Dutch cabinet sent last week, has now been invalidated by Trump himself. And why are we doing this? Convenience. But it is very scary to make yourself 100% dependent on the goodwill of the American government merely because it is convenient. So let’s not. [1]

1. https://berthub.eu/articles/posts/you-can-no-longer-base-your-government-and-society-on-us-clouds/

Tags

#Data,#Privacy,#CloudServices,#DigitalSovereignty.#EU,#GDPR

---

Mastodon	Peertube	Join Mastodon

AI statement : Consent is NOT granted to use the content of this blog for the purposes of AI training or similar activity. Consent CANNOT be assumed, it has to be granted.

Data Detox workshops 2024

Now that the library IT room is being updated, it would be a good time to look at trying to actually present some of the Data Detox workshops as listed below.

• Your Data Detox Starts Here
• Demystifying Your Data
• Declutter Your Phone with an App Cleanse
• Smartphones, Smart Habits

• Facts vs. Feelings: Keep Calm and Spot the Design Tricks

  Data Detox Workshops

Tags

#Data,#Detox,#Workshop,#DataDetox,#StemGroup,#CodeClub

---

Mastodon	Peertube	Join Mastodon

AI statement : Consent is NOT granted to use the content of this blog for the purposes of AI training or similar activity. Consent CANNOT be assumed, it has to be granted.

Zoom can be used to train AI

Looks like the new Zoom terms and conditions now allow training AI on user content with no opt out. 🤣 See section 10.2

I am not too sure what data is shared, or how it is used, but big tech is clearly incapable of playing fairly, we have companies using all uploaded data to train AI, for example photos, text, code anything and usually the consent is given via terms and conditions so end users are so tied in to closed systems, the users auto consent weather they like it or not.

The fact that zoom is used by organisation such as child protection agencies also for me RAISES SERIOUS CONCERNS.

Please consider using an alternative that actually protects and respects privacy such as Big Blue Button

Tags

#Zoom,#Privacy,#Data,#AI,#Replacement,#GDPR,#BigBlueButton.

---

Mastodon	Peertube	Join Mastodon

AI statement : Consent is NOT granted to use the content of this blog for the purposes of AI training or similar activity. Consent CANNOT be assumed, it has to be granted.