{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Manipulate text files in bash\n", "\n", "---\n", "[Recorded lecture: 1:36.15 - 2:19:00](https://youtu.be/IDfFa3ulC90)\n", "\n", "---\n", "\n", "The bash language has several commands to read and manipulate txt files, such as: head, tail, more, less, sort, join, wc, uniq. Here we are going to use some of them.\n", "\n", " cd /media/sf_LVM_shared/my_SE_data/exercise\n", " jupyter-notebook bashinter_osgeo.ipynb\n", "\n", "\n", "## Pattern matching\n", "\n", "Create a little file from a large file:" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [], "source": [ "! head -1000 txt/aver_month_nuts3_fire.asc > input.txt" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Read/explore the input.txt file" ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "scrolled": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "NUTS YYYY MM 0 BAREA\n", "BG311 2005 04 2 0.282594\n", "BG311 2006 11 2 0.600812\n", "BG311 2007 01 3 65.8331\n", "BG311 2007 02 3 9.78246\n", "BG311 2007 04 2 44.4997\n", "BG311 2007 06 2 30.5861\n", "BG311 2007 07 2 5534.21\n", "BG312 2005 04 3 10.6419\n", "BG312 2006 10 2 0.293182\n" ] } ], "source": [ "! head input.txt" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "DE425 2000 06 4 3.1973\n", "DE425 2000 07 4 0.724873\n", "DE425 2000 08 4 4.67528\n", "DE425 2000 09 3 0.194243\n", "DE425 2001 04 2 0.0724194\n", "DE425 2001 05 2 0.66708\n", "DE425 2001 08 2 0.0421668\n", "DE425 2002 02 2 0.0125149\n", "DE425 2002 03 2 0.492932\n", "DE425 2002 04 4 1.06466\n" ] } ], "source": [ "! tail input.txt" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Count the line/word/character in a input.txt" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " 1000 5000 24535 input.txt\n" ] } ], "source": [ "! wc input.txt" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Search for a word in a file" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "BG311 2007 01 3 65.8331\n", "BG311 2007 02 3 9.78246\n", "BG311 2007 04 2 44.4997\n", "BG311 2007 06 2 30.5861\n", "BG311 2007 07 2 5534.21\n", "BG312 2007 01 3 114.535\n", "BG312 2007 02 3 17.3247\n", "BG312 2007 03 3 322.063\n", "BG312 2007 04 3 521.189\n", "BG312 2007 05 3 4.13178\n" ] } ], "source": [ "%%bash\n", "grep \"2007\" input.txt | head " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Sorting a file\n", "I want to search for a command able to sort the input.txt table based on the Year column (YYYY)." ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "apt-sortpkgs (1) - Utility to sort package index files\n", "bunzip2 (1) - a block-sorting file compressor, v1.0.6\n", "bzip2 (1) - a block-sorting file compressor, v1.0.6\n", "comm (1) - compare two sorted files line by line\n", "heapsort (3bsd) - sort functions\n", "mergesort (3bsd) - sort functions\n", "osmium-merge (1) - merge several sorted OSM files into one\n", "osmium-sort (1) - sort OSM files\n", "otbcli_ConvertSensorToGeoPoint (1) - OTB ConvertSensorToGeoPoint application\n", "otbgui_ConvertSensorToGeoPoint (1) - OTB ConvertSensorToGeoPoint application\n", "radixsort (3bsd) - radix sort\n", "sort (1) - sort lines of text files\n", "sortshp (1) - sort a Shape data set\n", "sradixsort (3bsd) - radix sort\n", "tsort (1) - perform topological sort\n" ] } ], "source": [ "! man -k sort" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "One of the last lines contain:\n", "sort (1) - sort lines of text files\n", "So i will search how to use the sort command:" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "SORT(1) User Commands SORT(1)\n", "\n", "NAME\n", " sort - sort lines of text files\n", "\n", "SYNOPSIS\n", " sort [OPTION]... [FILE]...\n", " sort [OPTION]... --files0-from=F\n", "\n", "DESCRIPTION\n", " Write sorted concatenation of all FILE(s) to standard output.\n", "\n", " With no FILE, or when FILE is -, read standard input.\n", "\n", " Mandatory arguments to long options are mandatory for short options\n", " too. Ordering options:\n", "\n", " -b, --ignore-leading-blanks\n", " ignore leading blanks\n", "\n", " -d, --dictionary-order\n", " consider only blanks and alphanumeric characters\n", "\n", " -f, --ignore-case\n", " fold lower case to upper case characters\n", "\n", " -g, --general-numeric-sort\n", " compare according to general numerical value\n", "\n", " -i, --ignore-nonprinting\n", " consider only printable characters\n", "\n", " -M, --month-sort\n", " compare (unknown) < 'JAN' < ... < 'DEC'\n", "\n", " -h, --human-numeric-sort\n", " compare human readable numbers (e.g., 2K 1G)\n", "\n", " -n, --numeric-sort\n", " compare according to string numerical value\n", "\n", " -R, --random-sort\n", " shuffle, but group identical keys. See shuf(1)\n", "\n", " --random-source=FILE\n", " get random bytes from FILE\n", "\n", " -r, --reverse\n", " reverse the result of comparisons\n", "\n", " --sort=WORD\n", " sort according to WORD: general-numeric -g, human-numeric -h,\n", " month -M, numeric -n, random -R, version -V\n", "\n", " -V, --version-sort\n", " natural sort of (version) numbers within text\n", "\n", " Other options:\n", "\n", " --batch-size=NMERGE\n", " merge at most NMERGE inputs at once; for more use temp files\n", "\n", " -c, --check, --check=diagnose-first\n", " check for sorted input; do not sort\n", "\n", " -C, --check=quiet, --check=silent\n", " like -c, but do not report first bad line\n", "\n", " --compress-program=PROG\n", " compress temporaries with PROG; decompress them with PROG -d\n", "\n", " --debug\n", " annotate the part of the line used to sort, and warn about ques‐\n", " tionable usage to stderr\n", "\n", " --files0-from=F\n", " read input from the files specified by NUL-terminated names in\n", " file F; If F is - then read names from standard input\n", "\n", " -k, --key=KEYDEF\n", " sort via a key; KEYDEF gives location and type\n", "\n", " -m, --merge\n", " merge already sorted files; do not sort\n", "\n", " -o, --output=FILE\n", " write result to FILE instead of standard output\n", "\n", " -s, --stable\n", " stabilize sort by disabling last-resort comparison\n", "\n", " -S, --buffer-size=SIZE\n", " use SIZE for main memory buffer\n", "\n", " -t, --field-separator=SEP\n", " use SEP instead of non-blank to blank transition\n", "\n", " -T, --temporary-directory=DIR\n", " use DIR for temporaries, not $TMPDIR or /tmp; multiple options\n", " specify multiple directories\n", "\n", " --parallel=N\n", " change the number of sorts run concurrently to N\n", "\n", " -u, --unique\n", " with -c, check for strict ordering; without -c, output only the\n", " first of an equal run\n", "\n", " -z, --zero-terminated\n", " line delimiter is NUL, not newline\n", "\n", " --help display this help and exit\n", "\n", " --version\n", " output version information and exit\n", "\n", " KEYDEF is F[.C][OPTS][,F[.C][OPTS]] for start and stop position, where\n", " F is a field number and C a character position in the field; both are\n", " origin 1, and the stop position defaults to the line's end. If neither\n", " -t nor -b is in effect, characters in a field are counted from the\n", " beginning of the preceding whitespace. OPTS is one or more single-let‐\n", " ter ordering options [bdfgiMhnRrV], which override global ordering\n", " options for that key. If no key is given, use the entire line as the\n", " key. Use --debug to diagnose incorrect key usage.\n", "\n", " SIZE may be followed by the following multiplicative suffixes: % 1% of\n", " memory, b 1, K 1024 (default), and so on for M, G, T, P, E, Z, Y.\n", "\n", " *** WARNING *** The locale specified by the environment affects sort\n", " order. Set LC_ALL=C to get the traditional sort order that uses native\n", " byte values.\n", "\n", "AUTHOR\n", " Written by Mike Haertel and Paul Eggert.\n", "\n", "REPORTING BUGS\n", " GNU coreutils online help: \n", " Report sort translation bugs to \n", "\n", "COPYRIGHT\n", " Copyright © 2017 Free Software Foundation, Inc. License GPLv3+: GNU\n", " GPL version 3 or later .\n", " This is free software: you are free to change and redistribute it.\n", " There is NO WARRANTY, to the extent permitted by law.\n", "\n", "SEE ALSO\n", " shuf(1), uniq(1)\n", "\n", " Full documentation at: \n", " or available locally via: info '(coreutils) sort invocation'\n", "\n", "GNU coreutils 8.28 January 2018 SORT(1)\n" ] } ], "source": [ "! man sort" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The -k option identify the column of sorting:\\\n", "Sorting based on column number 2 ( -k 2,2)\\\n", "sorting based on column number 2 and then number 1 ( -k 2,1)\\\n", "See again man sort for more options like -n -g\\\n", "Alfa numeric sorting:" ] }, { "cell_type": "code", "execution_count": 12, "metadata": { "scrolled": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "DE121 1997 05 2 0.232016\n", "DE122 1997 05 2 0.0637817\n", "DE124 1997 03 2 1.28501\n", "DE125 1997 05 2 0.107349\n", "DE128 1997 04 2 0.340913\n", "DE129 1997 03 2 0.297982\n", "DE12A 1997 03 2 0.0815152\n", "DE123 1998 04 2 0.434829\n", "DE124 1998 03 2 0.0796515\n", "DE12A 1998 05 2 0.345525\n", "sort: write failed: 'standard output': Broken pipe\n", "sort: write error\n" ] } ], "source": [ "! sort -k 2,2 input.txt | head " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "General numerical sorting" ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "NUTS YYYY MM 0 BAREA\n", "DE121 1997 05 2 0.232016\n", "DE122 1997 05 2 0.0637817\n", "DE124 1997 03 2 1.28501\n", "DE125 1997 05 2 0.107349\n", "DE128 1997 04 2 0.340913\n", "DE129 1997 03 2 0.297982\n", "DE12A 1997 03 2 0.0815152\n", "DE123 1998 04 2 0.434829\n", "DE124 1998 03 2 0.0796515\n", "sort: write failed: 'standard output': Broken pipe\n", "sort: write error\n" ] } ], "source": [ "! sort -k 2,2 -g input.txt | head " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "String numerical sorting" ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "NUTS YYYY MM 0 BAREA\n", "DE121 1997 05 2 0.232016\n", "DE122 1997 05 2 0.0637817\n", "DE124 1997 03 2 1.28501\n", "DE125 1997 05 2 0.107349\n", "DE128 1997 04 2 0.340913\n", "DE129 1997 03 2 0.297982\n", "DE12A 1997 03 2 0.0815152\n", "DE123 1998 04 2 0.434829\n", "DE124 1998 03 2 0.0796515\n", "sort: write failed: 'standard output': Broken pipe\n", "sort: write error\n" ] } ], "source": [ "! sort -k 2,2 -n input.txt | head " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Save the result of a command in a file by \"\">\"\" symbol" ] }, { "cell_type": "code", "execution_count": 15, "metadata": { "scrolled": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "1000 input_s.txt\n" ] } ], "source": [ "! sort -k 2,2 -g input.txt > input_s.txt\n", "! wc -l input_s.txt" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Which is the first and last year of observations?" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Append the command result to a file\n", "Add the result of a command in the already existing \"output\" file by '>>' symbol" ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "2000 input_s.txt\n" ] } ], "source": [ "! sort -k 3,3 -g input.txt >> input_s.txt\n", "! wc -l input_s.txt" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Use the variable\n", "Define the value of the variable, print it by putting it in front of the $ symbol" ] }, { "cell_type": "code", "execution_count": 18, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n" ] } ], "source": [ "! var=21 \n", "! echo $var" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Define the value of the variable using the result of a command" ] }, { "cell_type": "code", "execution_count": 19, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "189\n" ] } ], "source": [ "%%bash\n", "var=$(grep \"2007\" input.txt | wc -l)\n", "echo $var" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## For loop\n", "In computer science, a for-loop (or simply for loop) is a control flow statement for specifying iteration, which allows code to be executed repeatedly (source https://en.wikipedia.org/wiki/For_loop).\n", "We want to automatically count how many observations exist in the years 2007, 2006 and 2005 in the input.txt file.\n", "To solve this task we can use the variable and list word/number loop function" ] }, { "cell_type": "code", "execution_count": 21, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "BG311 2005 04 2 0.282594\n", "BG312 2005 04 3 10.6419\n", "BG313 2005 03 2 48.0927\n", "BG314 2005 03 4 2.21985\n", "BG315 2005 03 2 125.772\n", "BG315 2005 04 2 95.5232\n", "BG315 2005 06 2 3.70607\n", "BG321 2005 03 3 59.201\n", "BG321 2005 04 2 0.562725\n", "BG322 2005 03 3 6.33855\n" ] } ], "source": [ "%%bash \n", "for var in 2005 2006 2007; do\n", " grep $var input.txt \n", "done | head " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "and now we count" ] }, { "cell_type": "code", "execution_count": 22, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "121\n", "280\n", "189\n" ] } ], "source": [ "%%bash \n", "for var in 2005 2006 2007; do\n", " grep $var input.txt | wc -l \n", "done " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now we want to automatically count and save in a file how many observations exist from year 2000 to 2008 in input.txt file. For this use the serial number list loop function." ] }, { "cell_type": "code", "execution_count": 24, "metadata": {}, "outputs": [], "source": [ "%%bash \n", "rm -f input_wc.txt\n", "for ((var=2000 ; var<=2008 ; var++)); do\n", " grep $var input.txt | wc -l >> input_wc.txt \n", "done " ] }, { "cell_type": "code", "execution_count": 25, "metadata": { "scrolled": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "62\n", "34\n", "48\n", "93\n", "46\n", "121\n", "280\n", "189\n", "2\n" ] } ], "source": [ "! head input_wc.txt" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## If condition in a for loop\n", "As for the previews exercise, we want to automatically count how many observations exist from year 2000 to 2008 in input.txt file, but not for the year 2003. For this you should use the serial number list loop function with the if condition." ] }, { "cell_type": "code", "execution_count": 26, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "2778\n", "2643\n", "2641\n", "2894\n", "2837\n", "3011\n", "775\n", "0\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "rm: cannot remove 'no2003output.txt': No such file or directory\n" ] } ], "source": [ "%%bash\n", "rm no2003output.txt\n", "for ((year=2000 ; year<=2008 ; year++)); do\n", "if [ $year != 2003 ] ; then\n", " grep \" $year \" txt/aver_month_nuts3_fire.asc | wc -l >> no2003output.txt \n", "fi\n", "done\n", "cat no2003output.txt" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The same operation can be done by saving the file outised the for loop. \n", "When to use the first approch and when to use the second one?" ] }, { "cell_type": "code", "execution_count": 29, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "2778\n", "2643\n", "2641\n", "2894\n", "2837\n", "3011\n", "775\n", "0\n" ] } ], "source": [ "%%bash\n", "rm no2003output.txt\n", "for ((year=2000 ; year<=2008 ; year++)); do\n", "if [ $year != 2003 ] ; then\n", " grep \" $year \" txt/aver_month_nuts3_fire.asc | wc -l \n", "fi\n", "done > no2003output.txt \n", "cat no2003output.txt" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "I need to know in each year which was the biggest fire and print it. I can use the sort command and get the largest fire in the last position." ] }, { "cell_type": "code", "execution_count": 30, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "GR253 2000 07 3 23216.3\n", "PT117 2001 08 464 7226.27\n", "PT118 2002 08 448 14574.7\n", "PT150 2004 07 5 16599\n", "PT164 2005 08 114 41830.3\n", "ES114 2006 08 554 52093.4\n", "BG422 2007 08 4 12972.6\n" ] } ], "source": [ "%%bash\n", "for ((year=2000 ; year<=2008 ; year++)); do\n", "if [ $year != 2003 ] ; then\n", " grep \" $year \" txt/aver_month_nuts3_fire.asc | sort -k 5,5 -g | tail -1 \n", "fi\n", "done" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Exercise**\n", "\n", "Perform the same loop but excluding the year from 2002 to 2004.\n", "Use the \"man test\" to see the option for the if condition.\n", "Googled \"if statement with multiple condition bash\"." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Checking the flow statement\n", "How can I check that the results are correct and that i'm using the correct variables? \n", "By printing the variable during the process and if you need also in the file." ] }, { "cell_type": "code", "execution_count": 31, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "processing year 2000\n", "processing year 2001\n", "processing year 2002\n", "processing year 2004\n", "processing year 2005\n", "processing year 2006\n", "processing year 2007\n", "processing year 2008\n" ] } ], "source": [ "%%bash\n", "rm no2003output.txt\n", "for ((year=2000 ; year<=2008 ; year++)); do\n", "if [ $year != 2003 ] ; then\n", " echo processing year $year \n", " grep \" $year \" txt/aver_month_nuts3_fire.asc | wc -l >> no2003output.txt \n", "fi\n", "done\n" ] }, { "cell_type": "code", "execution_count": 33, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "2778\n", "2643\n", "2641\n", "2894\n", "2837\n", "3011\n", "775\n", "0\n" ] } ], "source": [ "! head no2003output.txt" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "I can also run manually a command and compare the results." ] }, { "cell_type": "code", "execution_count": 34, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "775\n", "2641\n" ] } ], "source": [ "! grep \" 2007 \" txt/aver_month_nuts3_fire.asc | wc -l \n", "! grep \" 2002 \" txt/aver_month_nuts3_fire.asc | wc -l " ] }, { "cell_type": "code", "execution_count": 35, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "year 2000\n", " 2778 13890 67910\n", "year 2001\n", " 2643 13215 64585\n", "year 2002\n", " 2641 13205 64534\n", "year 2004\n", " 2894 14470 70924\n", "year 2005\n", " 2837 14185 69493\n", "year 2006\n", " 3011 15055 73972\n", "year 2007\n", " 775 3875 19026\n", "year 2008\n", " 0 0 0\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\n", "real\t0m0.132s\n", "user\t0m0.021s\n", "sys\t0m0.036s\n" ] } ], "source": [ "%%bash\n", "time for ((year=2000 ; year<=2008 ; year++)); do\n", "if [ $year != 2003 ] ; then\n", " echo year $year \n", " grep \" $year \" txt/aver_month_nuts3_fire.asc | wc \n", "fi\n", "done" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Debugging\n", "The shell reports message and status symbols in case of error syntax, incorrect commands or inexistent files.\n", "Here are reported the most common errors using the example:" ] }, { "cell_type": "code", "execution_count": 36, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "2778\n", "2643\n", "2641\n", "3078\n", "2894\n", "2837\n", "3011\n", "775\n", "0\n" ] } ], "source": [ "%%bash\n", "for ((year=2000 ; year<=2008 ; year++)); do\n", " grep \" $year \" txt/aver_month_nuts3_fire.asc | wc -l \n", "done" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Run the script and see the error results.\n", "\n", "The loop was not close and after the bash error a series of no sense python errors are reported." ] }, { "cell_type": "code", "execution_count": 37, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "bash: line 3: syntax error: unexpected end of file\n" ] }, { "ename": "CalledProcessError", "evalue": "Command 'b'for ((year=2000 ; year<=2008 ; year++)); do\\n grep \" $year \" txt/aver_month_nuts3_fire.asc | wc -l\\n'' returned non-zero exit status 2.", "output_type": "error", "traceback": [ "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", "\u001b[0;31mCalledProcessError\u001b[0m Traceback (most recent call last)", "\u001b[0;32m\u001b[0m in \u001b[0;36m\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mget_ipython\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrun_cell_magic\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'bash'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m''\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'for ((year=2000 ; year<=2008 ; year++)); do\\n grep \" $year \" txt/aver_month_nuts3_fire.asc | wc -l\\n'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m", "\u001b[0;32m~/miniconda3/lib/python3.8/site-packages/IPython/core/interactiveshell.py\u001b[0m in \u001b[0;36mrun_cell_magic\u001b[0;34m(self, magic_name, line, cell)\u001b[0m\n\u001b[1;32m 2397\u001b[0m \u001b[0;32mwith\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbuiltin_trap\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 2398\u001b[0m \u001b[0margs\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mmagic_arg_s\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcell\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 2399\u001b[0;31m \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mfn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 2400\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mresult\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 2401\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;32m~/miniconda3/lib/python3.8/site-packages/IPython/core/magics/script.py\u001b[0m in \u001b[0;36mnamed_script_magic\u001b[0;34m(line, cell)\u001b[0m\n\u001b[1;32m 140\u001b[0m \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 141\u001b[0m \u001b[0mline\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mscript\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 142\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mshebang\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mline\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcell\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 143\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 144\u001b[0m \u001b[0;31m# write a basic docstring:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;32m~/miniconda3/lib/python3.8/site-packages/decorator.py\u001b[0m in \u001b[0;36mfun\u001b[0;34m(*args, **kw)\u001b[0m\n\u001b[1;32m 230\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mkwsyntax\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 231\u001b[0m \u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkw\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mfix\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkw\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0msig\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 232\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mcaller\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfunc\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mextras\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0margs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkw\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 233\u001b[0m \u001b[0mfun\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__name__\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mfunc\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__name__\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 234\u001b[0m \u001b[0mfun\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__signature__\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0msig\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;32m~/miniconda3/lib/python3.8/site-packages/IPython/core/magic.py\u001b[0m in \u001b[0;36m\u001b[0;34m(f, *a, **k)\u001b[0m\n\u001b[1;32m 185\u001b[0m \u001b[0;31m# but it's overkill for just that one bit of state.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 186\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mmagic_deco\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0marg\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 187\u001b[0;31m \u001b[0mcall\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mlambda\u001b[0m \u001b[0mf\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0ma\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mk\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mf\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0ma\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mk\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 188\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 189\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mcallable\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0marg\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;32m~/miniconda3/lib/python3.8/site-packages/IPython/core/magics/script.py\u001b[0m in \u001b[0;36mshebang\u001b[0;34m(self, line, cell)\u001b[0m\n\u001b[1;32m 243\u001b[0m \u001b[0msys\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstderr\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mflush\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 244\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0margs\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mraise_error\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0mp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mreturncode\u001b[0m\u001b[0;34m!=\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 245\u001b[0;31m \u001b[0;32mraise\u001b[0m \u001b[0mCalledProcessError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mreturncode\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcell\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0moutput\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mout\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mstderr\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0merr\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 246\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 247\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0m_run_script\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mp\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcell\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mto_close\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;31mCalledProcessError\u001b[0m: Command 'b'for ((year=2000 ; year<=2008 ; year++)); do\\n grep \" $year \" txt/aver_month_nuts3_fire.asc | wc -l\\n'' returned non-zero exit status 2." ] } ], "source": [ "%%bash\n", "for ((year=2000 ; year<=2008 ; year++)); do\n", " grep \" $year \" txt/aver_month_nuts3_fire.asc | wc -l" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Bash: syntax error near unexpected token `('\n", "\n", "The error is near the brackets. Often it is just a space or a bracket that has not been closed" ] }, { "cell_type": "code", "execution_count": 38, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "bash: line 1: syntax error near unexpected token `('\n", "bash: line 1: `for ( (year=2000 ; year<=2008 ; year++)); do'\n" ] }, { "ename": "CalledProcessError", "evalue": "Command 'b'for ( (year=2000 ; year<=2008 ; year++)); do\\n grep \" $year \" txt/aver_month_nuts3_fire.asc | wc -l \\ndone \\n'' returned non-zero exit status 2.", "output_type": "error", "traceback": [ "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", "\u001b[0;31mCalledProcessError\u001b[0m Traceback (most recent call last)", "\u001b[0;32m\u001b[0m in \u001b[0;36m\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mget_ipython\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrun_cell_magic\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'bash'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m''\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'for ( (year=2000 ; year<=2008 ; year++)); do\\n grep \" $year \" txt/aver_month_nuts3_fire.asc | wc -l \\ndone \\n'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m", "\u001b[0;32m~/miniconda3/lib/python3.8/site-packages/IPython/core/interactiveshell.py\u001b[0m in \u001b[0;36mrun_cell_magic\u001b[0;34m(self, magic_name, line, cell)\u001b[0m\n\u001b[1;32m 2397\u001b[0m \u001b[0;32mwith\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbuiltin_trap\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 2398\u001b[0m \u001b[0margs\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mmagic_arg_s\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcell\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 2399\u001b[0;31m \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mfn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 2400\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mresult\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 2401\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;32m~/miniconda3/lib/python3.8/site-packages/IPython/core/magics/script.py\u001b[0m in \u001b[0;36mnamed_script_magic\u001b[0;34m(line, cell)\u001b[0m\n\u001b[1;32m 140\u001b[0m \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 141\u001b[0m \u001b[0mline\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mscript\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 142\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mshebang\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mline\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcell\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 143\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 144\u001b[0m \u001b[0;31m# write a basic docstring:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;32m~/miniconda3/lib/python3.8/site-packages/decorator.py\u001b[0m in \u001b[0;36mfun\u001b[0;34m(*args, **kw)\u001b[0m\n\u001b[1;32m 230\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mkwsyntax\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 231\u001b[0m \u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkw\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mfix\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkw\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0msig\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 232\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mcaller\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfunc\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mextras\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0margs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkw\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 233\u001b[0m \u001b[0mfun\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__name__\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mfunc\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__name__\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 234\u001b[0m \u001b[0mfun\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__signature__\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0msig\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;32m~/miniconda3/lib/python3.8/site-packages/IPython/core/magic.py\u001b[0m in \u001b[0;36m\u001b[0;34m(f, *a, **k)\u001b[0m\n\u001b[1;32m 185\u001b[0m \u001b[0;31m# but it's overkill for just that one bit of state.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 186\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mmagic_deco\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0marg\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 187\u001b[0;31m \u001b[0mcall\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mlambda\u001b[0m \u001b[0mf\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0ma\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mk\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mf\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0ma\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mk\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 188\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 189\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mcallable\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0marg\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;32m~/miniconda3/lib/python3.8/site-packages/IPython/core/magics/script.py\u001b[0m in \u001b[0;36mshebang\u001b[0;34m(self, line, cell)\u001b[0m\n\u001b[1;32m 243\u001b[0m \u001b[0msys\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstderr\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mflush\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 244\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0margs\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mraise_error\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0mp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mreturncode\u001b[0m\u001b[0;34m!=\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 245\u001b[0;31m \u001b[0;32mraise\u001b[0m \u001b[0mCalledProcessError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mreturncode\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcell\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0moutput\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mout\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mstderr\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0merr\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 246\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 247\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0m_run_script\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mp\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcell\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mto_close\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;31mCalledProcessError\u001b[0m: Command 'b'for ( (year=2000 ; year<=2008 ; year++)); do\\n grep \" $year \" txt/aver_month_nuts3_fire.asc | wc -l \\ndone \\n'' returned non-zero exit status 2." ] } ], "source": [ "%%bash \n", "for ( (year=2000 ; year<=2008 ; year++)); do\n", " grep \" $year \" txt/aver_month_nuts3_fire.asc | wc -l \n", "done " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Bash command error: use \"man -k\" for searching the operation that you need." ] }, { "cell_type": "code", "execution_count": 39, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "0\n", "0\n", "0\n", "0\n", "0\n", "0\n", "0\n", "0\n", "0\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "bash: line 2: grap: command not found\n", "bash: line 2: grap: command not found\n", "bash: line 2: grap: command not found\n", "bash: line 2: grap: command not found\n", "bash: line 2: grap: command not found\n", "bash: line 2: grap: command not found\n", "bash: line 2: grap: command not found\n", "bash: line 2: grap: command not found\n", "bash: line 2: grap: command not found\n" ] } ], "source": [ "%%bash\n", "for ((year=2000 ; year<=2008 ; year++)); do\n", " grap \" $year \" txt/aver_month_nuts3_fire.asc | wc -l \n", "done" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Invalid command option: \"wc: invalid option -- 'k'\". Read carefully the manual for the wc command." ] }, { "cell_type": "code", "execution_count": 40, "metadata": {}, "outputs": [], "source": [ "import warnings; warnings.simplefilter('ignore')" ] }, { "cell_type": "code", "execution_count": 41, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "wc: invalid option -- 'k'\n", "Try 'wc --help' for more information.\n", "wc: invalid option -- 'k'\n", "Try 'wc --help' for more information.\n", "wc: invalid option -- 'k'\n", "Try 'wc --help' for more information.\n", "wc: invalid option -- 'k'\n", "Try 'wc --help' for more information.\n", "wc: invalid option -- 'k'\n", "Try 'wc --help' for more information.\n", "wc: invalid option -- 'k'\n", "Try 'wc --help' for more information.\n", "wc: invalid option -- 'k'\n", "Try 'wc --help' for more information.\n", "wc: invalid option -- 'k'\n", "Try 'wc --help' for more information.\n", "wc: invalid option -- 'k'\n", "Try 'wc --help' for more information.\n" ] }, { "ename": "CalledProcessError", "evalue": "Command 'b'for ((year=2000 ; year<=2008 ; year++)); do\\n grep \" $year \" txt/aver_month_nuts3_fire.asc | wc -k\\ndone\\n'' returned non-zero exit status 1.", "output_type": "error", "traceback": [ "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", "\u001b[0;31mCalledProcessError\u001b[0m Traceback (most recent call last)", "\u001b[0;32m\u001b[0m in \u001b[0;36m\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mget_ipython\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrun_cell_magic\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'bash'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m''\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'for ((year=2000 ; year<=2008 ; year++)); do\\n grep \" $year \" txt/aver_month_nuts3_fire.asc | wc -k\\ndone\\n'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m", "\u001b[0;32m~/miniconda3/lib/python3.8/site-packages/IPython/core/interactiveshell.py\u001b[0m in \u001b[0;36mrun_cell_magic\u001b[0;34m(self, magic_name, line, cell)\u001b[0m\n\u001b[1;32m 2397\u001b[0m \u001b[0;32mwith\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbuiltin_trap\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 2398\u001b[0m \u001b[0margs\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mmagic_arg_s\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcell\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 2399\u001b[0;31m \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mfn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 2400\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mresult\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 2401\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;32m~/miniconda3/lib/python3.8/site-packages/IPython/core/magics/script.py\u001b[0m in \u001b[0;36mnamed_script_magic\u001b[0;34m(line, cell)\u001b[0m\n\u001b[1;32m 140\u001b[0m \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 141\u001b[0m \u001b[0mline\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mscript\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 142\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mshebang\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mline\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcell\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 143\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 144\u001b[0m \u001b[0;31m# write a basic docstring:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;32m~/miniconda3/lib/python3.8/site-packages/decorator.py\u001b[0m in \u001b[0;36mfun\u001b[0;34m(*args, **kw)\u001b[0m\n\u001b[1;32m 230\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mkwsyntax\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 231\u001b[0m \u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkw\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mfix\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkw\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0msig\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 232\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mcaller\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfunc\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mextras\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0margs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkw\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 233\u001b[0m \u001b[0mfun\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__name__\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mfunc\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__name__\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 234\u001b[0m \u001b[0mfun\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__signature__\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0msig\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;32m~/miniconda3/lib/python3.8/site-packages/IPython/core/magic.py\u001b[0m in \u001b[0;36m\u001b[0;34m(f, *a, **k)\u001b[0m\n\u001b[1;32m 185\u001b[0m \u001b[0;31m# but it's overkill for just that one bit of state.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 186\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mmagic_deco\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0marg\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 187\u001b[0;31m \u001b[0mcall\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mlambda\u001b[0m \u001b[0mf\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0ma\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mk\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mf\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0ma\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mk\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 188\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 189\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mcallable\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0marg\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;32m~/miniconda3/lib/python3.8/site-packages/IPython/core/magics/script.py\u001b[0m in \u001b[0;36mshebang\u001b[0;34m(self, line, cell)\u001b[0m\n\u001b[1;32m 243\u001b[0m \u001b[0msys\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstderr\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mflush\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 244\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0margs\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mraise_error\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0mp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mreturncode\u001b[0m\u001b[0;34m!=\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 245\u001b[0;31m \u001b[0;32mraise\u001b[0m \u001b[0mCalledProcessError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mreturncode\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcell\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0moutput\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mout\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mstderr\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0merr\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 246\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 247\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0m_run_script\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mp\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcell\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mto_close\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;31mCalledProcessError\u001b[0m: Command 'b'for ((year=2000 ; year<=2008 ; year++)); do\\n grep \" $year \" txt/aver_month_nuts3_fire.asc | wc -k\\ndone\\n'' returned non-zero exit status 1." ] } ], "source": [ "%%bash\n", "for ((year=2000 ; year<=2008 ; year++)); do\n", " grep \" $year \" txt/aver_month_nuts3_fire.asc | wc -k\n", "done" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The file or directory does not exist: search for the correct file and directory, by using \"cd\" and \"pwd\"" ] }, { "cell_type": "code", "execution_count": 42, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "0\n", "0\n", "0\n", "0\n", "0\n", "0\n", "0\n", "0\n", "0\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "grep: ../aver_month_nuts3_fire.asc: No such file or directory\n", "grep: ../aver_month_nuts3_fire.asc: No such file or directory\n", "grep: ../aver_month_nuts3_fire.asc: No such file or directory\n", "grep: ../aver_month_nuts3_fire.asc: No such file or directory\n", "grep: ../aver_month_nuts3_fire.asc: No such file or directory\n", "grep: ../aver_month_nuts3_fire.asc: No such file or directory\n", "grep: ../aver_month_nuts3_fire.asc: No such file or directory\n", "grep: ../aver_month_nuts3_fire.asc: No such file or directory\n", "grep: ../aver_month_nuts3_fire.asc: No such file or directory\n" ] } ], "source": [ "%%bash\n", "for ((year=2000 ; year<=2008 ; year++)); do\n", " grep \" $year \" ../aver_month_nuts3_fire.asc | wc -l \n", "done" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**remove processed files**" ] }, { "cell_type": "code", "execution_count": 43, "metadata": {}, "outputs": [], "source": [ "! rm input_s.txt input.txt input_wc.txt no2003output.txt" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.10" } }, "nbformat": 4, "nbformat_minor": 4 }