Tuesday, July 5, 2022

Recent Questions - Unix & Linux Stack Exchange

Recent Questions - Unix & Linux Stack Exchange

Recent Questions - Unix & Linux Stack Exchange


Why tr redirected to dd unexpectedly cuts data stream?

Posted: 05 Jul 2022 02:13 PM PDT

I'm able to fill 1MB file with specific character like this:

> tr '\0' '#' </dev/zero | dd of=1MB.bin bs=1k count=1024  1024+0 records in  1024+0 records out  1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139671 s, 75.1 MB/s  

However if I use larger block size it ends-up with 64kB file:

> tr '\0' '#' </dev/zero | dd of=1MB.bin bs=1M count=1  0+1 records in  0+1 records out  65536 bytes (66 kB, 64 KiB) copied, 0.000240126 s, 273 MB/s  

Can anyone explain this behavior? Is it a buffering problem? I see it is not connected with /dev/zero special file. The same result with regular file.

WHY does awk -F work for most letters, but NOT for the letter "t"?

Posted: 05 Jul 2022 02:29 PM PDT

 July 2022      mac os Monterey V12.1      awk --version 20200816     GNU bash, version 3.2.57(1)-release (x86_64-apple-darwin21)  

WHY does awk -F work for most letters, but NOT for the letter "t"? I have the solution, but I would like to understand WHY awk fails for the letter "t"

echo " kount t's in total  <== FAILS"  echo "total" | awk -F "t" '{print NF-1}'    echo " kount o's in toot"  echo "toot"  | awk -F "o" '{print NF-1}'    echo " kount e's in tweet"  echo "tweet"  | awk -F "e" '{print NF-1}'    echo " use gsub(regexp, replacement [, target])  "  echo "toot" | awk '{print gsub(/t/, "")}'   

...

Where/How are the XDG_*_DIR user directory variables supposed to be set?

Posted: 05 Jul 2022 01:39 PM PDT

I'm referring to: https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html

I have the following files in my .config directory:

~/.config/user-dirs.dirs:

# ...  XDG_DOWNLOAD_DIR="$HOME/downloads"  # ... there are only comments in this file, except the line above ...  

(I guess this file needs to be sourced somewhere? But where and how?)

~/.config/user-dirs.conf:

enabled=False  

(This file seems to be necesseary to prevent xdg-user-dirs-update from overwriting ~/.config/user-dirs.dirs.)

However, $XDG_DOWNLOAD_DIR is not set when I run a shell.

Where/How are the XDG*DIR user directory variables supposed to be set?

Am I supposed to source ~/.config/user-dirs.dirs in one of the files sourced by my login shell (e.g. in .bash_profile)?

FFMPEG equivalent to Gstreamers "shmsink"?

Posted: 05 Jul 2022 12:25 PM PDT

I would like to replace this Gstreamer command:

gst-launch-1.0 -v filesrc location="VideoFile.mp4" qtdemux name=demux demux.video_0 h264parse v4l2h264dec v4l2convert 'video/x-raw,format=BGRA' shmsink socket-path=/tmp/feed2-control-pipe shm-size=81100800

with an ffmpeg equivalent such as:

ffmpeg -i "VideoFile.mp4" -f rawvideo -pix_fmt bgra pipe:1 > /tmp/feed2-control-pipe

I could not find a suitable answer to this question anywhere else. Thank you in advance.

In Linux what is the difference between swapping and paging?

Posted: 05 Jul 2022 12:24 PM PDT

I am trying to figure out what the difference is between paging and swapping on linux. My question mainly relates to the output vmstat gives you. Example:

  $ vmstat -s    8022500 K total memory    2761696 K used memory    3647052 K active memory    2316288 K inactive memory    1579228 K free memory     295964 K buffer memory    3385612 K swap cache    4194300 K total swap          0 K used swap    4194300 K free swap      15273 non-nice user cpu ticks          0 nice user cpu ticks       6141 system cpu ticks     198640 idle cpu ticks       1621 IO-wait cpu ticks          0 IRQ cpu ticks        282 softirq cpu ticks          0 stolen cpu ticks    2190475 pages paged in     882132 pages paged out          0 pages swapped in          0 pages swapped out    1542526 interrupts    3926900 CPU context switches  

I thought that in linux swap really means paging (anonymous paging). So why are there still different metrics? And assuming there is a difference, if a page is paged out (not swapped), where does go? Is it still written to the swap file?

I know heavy swap is an indicator of memory pressure, can the same be said about paging out? Can heavy page out ever be an indicator of memory issues? If so, what should I look for?

“systemctl start dhcpd.service” command not working for PXE server setup

Posted: 05 Jul 2022 11:40 AM PDT

I am trying to setup a PXE server on my laptop on CentOS 7, following the tutorial on: https://www.linuxtechi.com/configure-pxe-installation-server-centos-7/#comment-35567

All of the configuration files and setup procedures are from this website. On "Step: 6 Start and enable xinetd, dhcp, and vsftpd service.", The commands: "systemctl start xinetd" and "systemctl enable xinetd" work, but when I run the command: "systemctl start dhcpd.service", I receive the following error message:

Job for dhcpd.service failed because the control process exited with error code. See "systemctl status dhcpd.service" and "journalctl -xe" for details.  

When I run "systemctl status -l dhcpd.service", I receive the following error message:

systemctl status -l dhcpd.service   dhcpd.service - DHCPv4 Server Daemon     Loaded: loaded (/usr/lib/systemd/system/dhcpd.service; disabled; vendor preset: disabled)     Active: failed (Result: exit-code) since Tue 2022-07-05 11:18:07 EDT; 1min 12s ago       Docs: man:dhcpd(8)             man:dhcpd.conf(5)    Process: 11655 ExecStart=/usr/sbin/dhcpd -f -cf /etc/dhcp/dhcpd.conf -user dhcpd -group dhcpd --no-pid (code=exited, status=1/FAILURE)   Main PID: 11655 (code=exited, status=1/FAILURE)    Jul 05 11:18:07 localhost.localdomain dhcpd[11655]:    to which interface virbr0 is attached. **  Jul 05 11:18:07 localhost.localdomain dhcpd[11655]:   Jul 05 11:18:07 localhost.localdomain dhcpd[11655]:   Jul 05 11:18:07 localhost.localdomain dhcpd[11655]: No subnet declaration for enp0s20f0u13 (10.249.6.154).  Jul 05 11:18:07 localhost.localdomain dhcpd[11655]: ** Ignoring requests on enp0s20f0u13.  If this is not what  Jul 05 11:18:07 localhost.localdomain dhcpd[11655]:    you want, please write a subnet declaration  Jul 05 11:18:07 localhost.localdomain systemd[1]: dhcpd.service: main process exited, code=exited, status=1/FAILURE  Jul 05 11:18:07 localhost.localdomain systemd[1]: Failed to start DHCPv4 Server Daemon.  Jul 05 11:18:07 localhost.localdomain systemd[1]: Unit dhcpd.service entered failed state.  Jul 05 11:18:07 localhost.localdomain systemd[1]: dhcpd.service failed.  

Also here is the Dhcpd.conf file:

#  # DHCP Server Configuration file.  #   see /usr/share/doc/dhcp*/dhcpd.conf.example  #   see dhcpd.conf(5) man page  #  # DHCP Server Configuration file.  ddns-update-style interim;    ignore client-updates;    authoritative;    allow booting;    allow bootp;    allow unknown-clients;    # internal subnet for my DHCP Server    subnet 172.168.1.0 netmask 255.255.255.0 {    range 172.168.1.21 172.168.1.151;    option domain-name-servers 172.168.1.11;    option domain-name "pxe.example.com";    option routers 172.168.1.11;    option broadcast-address 172.168.1.255;    default-lease-time 600;    max-lease-time 7200;    # IP of PXE Server    next-server 172.168.1.11;    filename "pxelinux.0";    }  

What do I need to change in my dhcpd.conf file to make the command "systemctl start dhcpd.service" work so I can finish going through the PXE server tutorial?

Using RAID 1 to mirror two external backup drives

Posted: 05 Jul 2022 02:34 PM PDT

I am looking to learn the setup for RAID on my OS (Pop OS) and also backup my laptop. I want a proper backup scheme in place on one external drive (drive A) and I want a RAID 1 setup between another external drive (drive B). Neither drive A nor B will mirror my hard drive, but I would like them to mirror one another for redundancy in backups. I tried setting up RAID 1 for them, but they sought to mirror my boot drive, which isn't what I seek. Is RAID an appropriate tool for mirroring external drives in such a manner? Or is there a better tool? Do the drives have to be present at boot? I hit a bump with needing the drives at all times when rebooting the computer without the drives present. Thank you.

Zip all folders in current directory

Posted: 05 Jul 2022 12:24 PM PDT

Im new to shell scripting and I was wondering if there was a way to zip all subdirectories individually in the current directory doing something that seemed familiar to high level programers. ie:

for folder in currentDir:      zip(folder)  

GNU Recode - Properly decode mixed HTML Character/Numeric encoded text?

Posted: 05 Jul 2022 10:52 AM PDT

I recently found GNU recode as something that can be used to decode HTML entities, however when looking at a piece of malware I noticed that it appears to be mixed HTML character/entity encoding, such as this. I changed the URL from the malicious domain to example.com:

&#x68;&#x74;&#x74;&#x70;&#x73;&colon;&sol;&sol;&#x65;&#x78;&#x61;&#x6d;&#x70;&#x6c;&#x65;&period;&#x63;&#x6f;&#x6d;&sol;

When I pipe this into recode html..ascii, I get the following, which decodes the numeric entities but not the character ones.

https&colon;&sol;&sol;example&period;com&sol;

I'm not finding good docs on the formats handled by recode, is there a way I can have it do a second pass to decode the character entities also, or is there a better cli tool I can be using to decode this?

how to add a number as well as format change to column value and print it in 3rd column

Posted: 05 Jul 2022 01:01 PM PDT

I have these two columns with header:

CHR BP 1 78901

i need to add a character to the first column numbers (chr). I also need to make a third column in which one score being added to the second column value (x to x+1) in this way:

CHR BP new-column chr1 78901 78902

appreciate your helps

Assigning data extracted from a JSON document to a Zsh shell parameter

Posted: 05 Jul 2022 11:27 AM PDT

During everyday shell sessions, I very often find myself needing to assign data from a JSON document (extracted via some jq filter) to a Zsh shell parameter: JSON scalars to Zsh scalars, JSON arrays to Zsh arrays, and JSON objects to Zsh associative arrays. The problem—and what previously asked questions don't seem to tackle—is that the data often contains newlines (and even NUL bytes), making this a rather nontrivial task.

Here is what I have come up with so far:

function assign-from-json {      local -A opts && zparseopts -A opts -D -F -M -- a A && typeset -r opts      if [[ $# -ne 3 || ( -v opts[-a] && -v opts[-A] ) ]] ; then          >&2 printf 'Usage: %s [-a|-A] NAME FILTER JSON\n' $0          return 2      fi      if [[ -v opts[-a] ]] ; then          local -a lengths && { lengths=( "${(@f)$( jq -r "$2 | .[] | tostring | length" <<< $3 )}" ) || return $? } && typeset -r lengths          local data && { data="$( jq -j "$2 | .[] | tostring" <<< $3 )" || return $? } && typeset -r data          local elem          local -a elems          for length in "${lengths[@]}" ; do              read -u 0 -k $length elem              elems+=$elem          done <<< $data          eval "${(q)1}"='( "${elems[@]}" )'      elif [[ -v opts[-A] ]] ; then          local transformed_json && { transformed_json="$( jq "$2 | to_entries | map(.key, .value)" <<< $3 )" || return $? } && typeset -r transformed_json          assign-from-json -a $1 "." $transformed_json      else          eval "${(q)1}"="${(q)$( jq -r $2 <<< $3 )}"      fi  }  

In most cases it works quite well:

% json='  {      "scalar": "Hello, world",      "array": [1, 2, 3],      "scary_scalar": "\nNewlines\u0000NUL bytes\ttabs",      "scary_array": [          "A\nvery\u0000scary\nvalue",          "A less scary value",          "eh"      ]  }  '  % assign-from-json scalar '.scalar' $json && printf '%q\n' $scalar  Hello,\ world  % typeset -a array && assign-from-json -a array '.array' $json && printf '%q\n' "${array[@]}"  1  2  3  % assign-from-json scary_scalar '.scary_scalar' $json && printf '%q\n' $scary_scalar  $'\n'Newlines$'\0'NUL\ bytes$'\t'tabs  % typeset -a scary_array && assign-from-json -a scary_array '.scary_array' $json && printf '%q\n' "${scary_array[@]}"  A$'\n'very$'\0'scary$'\n'value  A\ less\ scary\ value  eh  % typeset -A assoc && assign-from-json -A assoc '.' $json && printf '%q -> %q\n' "${(@kv)assoc}"  array -> \[1,2,3\]  scary_array -> \[\"A\\nvery\\u0000scary\\nvalue\",\"A\ less\ scary\ value\",\"eh\"\]  scary_scalar -> $'\n'Newlines$'\0'NUL\ bytes$'\t'tabs  scalar -> Hello,\ world  

Unfortunately it seems to struggle with trailing newlines:

% assign-from-json bad_scalar '.' '"foo\n"' && printf '%q\n$ $bad_scalar  foo  # expected: foo$'\n'  
  1. I assume the problem with trailing newlines is due to command substitution removing them. Do you see an easy way to fix it?
  2. One can do assign-from-json -A assoc ... even if assoc is not typeset as an associative array. How can I prevent that from being possible?
  3. Do you see any other problems with the code?

How to give a chromium web app a proper title and icon?

Posted: 05 Jul 2022 10:19 AM PDT

I am on Wayland and Discord share screen did not work. So I made a .desktop file which launches Chromium with --app=https://discord.com/app with a few extra flags. However it launches it and it shows up in the dock like this:

enter image description here

How can I make it so that I can pin it as a Discord "app" so it does not show up in the unpinned apps area?

My .desktop file (chromium-freeworld is Fedora's functional chromium):

[Desktop Entry]  Name=Discord Web  Type=Application  Exec=chromium-freeworld --user-data-dir=/home/x/.config/discordapp --profile-directory=DiscordApp --app=https://discord.com/app  Terminal=false  Icon=/home/x/.config/discordapp/discord.png  

Would I be able to access my Windows files if I am to make a dual boot into Linux?

Posted: 05 Jul 2022 10:19 AM PDT

So, I wish to dual install Linux into my computer. My concern is suppose there is a program or file which is openable both in Linux and in Windows (eg: text md file), then is there any way to access the windows files from a Linux boot?

Btw I plan on using Fedora.

Cant seem to access associative array with named reference in bash

Posted: 05 Jul 2022 11:32 AM PDT

# SETUP PHP81 SYMLINKS  declare -A cgi=([path]="/opt/remi/php81/root/usr/bin/php-cgi" [filename]="php-cgi81")  declare -A config=([path]="/opt/remi/php81/root/usr/bin/php-config" [filename]="php-config81")  declare -A phpize=([path]="/opt/remi/php81/root/usr/bin/phpize" [filename]="phpize81")  declare -A pecl=([path]="/opt/remi/php81/root/usr/bin/pecl" [filename]="pecl81")  declare -A pear=([path]="/opt/remi/php81/root/usr/bin/pear" [filename]="pear81")    declare -a symlinks=("cgi" "config" "phpize" "pecl" "pear")    echo -en "[INFO]: Setting up PHP 8.1 symlinks\n"  cd /usr/bin    for symlink in "${symlinks[@]}";  do      echo -en "[INFO]: Creating symlink for ${symlink}\n"        # Get the array that matches the string in symlink      props="$symlink[@]"      # Print out the filename property eg cgi[filename], config[filename], phpize[filename]      echo ${!props[filename]}              echo -en "\n"    done  

There is the section of code giving me issues. It currently outputs

./upgrade.sh   [INFO]: Installing php81 packages  [INFO]: Setting up PHP 8.1 symlinks  [INFO]: Creating symlink for cgi  php-cgi81 /opt/remi/php81/root/usr/bin/php-cgi    [INFO]: Creating symlink for config  php-config81 /opt/remi/php81/root/usr/bin/php-config    [INFO]: Creating symlink for phpize  phpize81 /opt/remi/php81/root/usr/bin/phpize    [INFO]: Creating symlink for pecl  pecl81 /opt/remi/php81/root/usr/bin/pecl    [INFO]: Creating symlink for pear  pear81 /opt/remi/php81/root/usr/bin/pear    The expected output is   ./upgrade.sh   [INFO]: Installing php81 packages  [INFO]: Setting up PHP 8.1 symlinks  [INFO]: Creating symlink for cgi  php-cgi8=    [INFO]: Creating symlink for config  php-config81    [INFO]: Creating symlink for phpize  phpize81    [INFO]: Creating symlink for pecl  pecl81    [INFO]: Creating symlink for pear  pear81  

How to calculate nodes, edges and degrees of TXT file using LINUX | UNIX (GITBASH)?

Posted: 05 Jul 2022 11:01 AM PDT

I am building a network by Web-Crawling of txt file extracted from WIKIPEDIA, Using UNIX commands (the tool is GitBash).

Already found the first lines of the file but need to count the number of nodes, edged and in & out degrees of the nodes.

Tried:

(Count the number of lines, words, and characters in the file)

wc file  47000 94000 72319473 file  

also, for each column, counted the entries that do not start with [Not A]:

awk -F'\t' 'NR==1{for (i=1;i<=NF;i++)h[i]=$i;next} {for (i=1;i<=NF;i++)c[i]+=($$i;next} {for (i=1;i<=NF;i++)c[i]+=($i !~ /[[]Not A/)} END{for (i=1;i<=NF;i++)print h[i],c[i]}' file:    0 46999 1,2,3,4,5,6,7,8,0,9,10,11,12,13,14,15,16 46999  

I am not sure this is what I need -count the number of nodes, edged and in & out degrees of the nodes Any suggestions?

Head of the file:

0 1,2,3,4,5,6,7,8,0,9,10,11,12,13,14,15,16 4 0,2,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,128,129,130,131,132,133,134,135,136,137,138,139,140,141,142,143,144,145,146,147,148,149,150,151,152,153,154,155,156,157,158,159,160,161,162,163,164,165,166,167,168,169,170,171,172,173,174,175,176,177,178,179,180,181,182,183,184,185,186,187,188,189,190,191,192,193,194,195,196,197,198,199,200,201,202,203,204,205,206,207,208,209,210,211,212,213,214,215,216,217,218,219,220,221,222,223,224,225,226,227,228,229,230,231,232,233,234,235,236,237,238,239,240,241,242,243,244,245,246,247,248,249,250,251,252,253,254,255,256,257,258,259,260,261,262,263,264,265,266,267,268,269,270,271,272,273,274,275,276,277,278,279,280,281,282,283,284,285,286,287,288,289,290,291,292,293,294,295,296,297,298,299,300,301,302,303,304,305,306,307,308,309,310,311,312,313,314,315,316,317,318,319,320,321,322,323,324,325,326,327,328,329,330,331,332,333,334,335,336,337,338,339,340,341,342,343,344,345,346,347,348,349,350,351,352,353,354,355,356,357,358,359,360,361,362,363,364,365,366,367,368,369,370,371,372,373,374,375,376,377,378,379,380,381,382,383,384,385,386,387,388,389,390,391,392,393,394,395,396,397,398,399,400,401,402,403,404,405,406,407,408,409,410,411,412,413,414,415,416,417,418,419,420,421,422,423,424,425,426,427,428,429,430,431,432,433,434,435,436,437,438,439,440,441,442,443,444,445,446,447,448,449,450,451,452,453,454,455,1,456,457,8,4,458,9,10,11,12,13,14,15,16 19 459,173,25,460,75,461,462,463,464,465,466,467,468,469,470,471,178,366,51,300,472,249,473,251,474,475,476,477,478,479,480,481,482,483,484,485,486,487,488,489,490,491,492,493,494,495,496,497,498,499,500,501,502,503,504,505,506,507,508,509,510,511,512,513,514,515,516,517,518,519,1,8,19,9,10,11,12,13,14,15,16 48 40,520,49,521,522,523,45,524,525,526,527,528,529,530,531,532,533,534,411,535,536,537,538,539,540,541,46,542,335,543,367,544,545,249,250,254,247,1,8,48,546,9,10,11,12,13,14,15,16 49 40,441,520,547,548,445,269,549,550,551,552,533,553,554,523,555,556,557,558,559,560,561,562,563,564,565,394,566,567,379,568,569,570,48,571,146,572,573,574,575,39,247,264,249,261,576,1,577,8,49,578,9,10,11,12,13,14,15,16 46 520,47,50,579,580,581,582,583,521,584,585,586,587,588,589,590,591,592,593,594,595,596,597,598,599,600,601,602,603,604,605,606,607,608,609,610,20,611,48,524,367,552,45,612,613,614,615,616,617,22,51,178,225,372,618,619,620,621,71,622,138,623,624,625,626,335,90,627,628,629,210,630,631,632,633,634,635,636,637,638,146,639,640,641,642,643,644,249,250,251,247,645,261,646,647,648,649,650,651,652,653,201,654,655,656,657,658,659,660,661,202,662,663,664,665,666,667,668,669,670,671,672,673,674,675,676,677,678,679,680,681,682,683,684,685,686,687,688,689,690,691,692,693,694,695,696,697,698,699,700,701,702,703,238,704,705,1,706,707,8,46,708,9,10,11,12,13,14,15,16 22 709,710,26,308,711,608,24,712,713,714,715,219,716,94,717,718,719,309,720,721,2,722,246,723,724,220,725,726,727,247,728,729,249,261,255,254,250,730,731,251,732,733,206,734,735,736,737,738,739,307,740,580,112,741,310,742,743,744,745,746,747,748,749,29,119,288,114,391,21,365,364,750,227,751,752,753,754,755,756,270,4,33,31,271,272,273,274,275,276,277,278,279,280,281,32,282,283,284,285,286,287,289,290,291,18,292,293,294,295,296,297,298,299,300,301,302,303,165,39,304,305,148,306,42,311,312,100,313,28,101,314,95,315,316,317,318,319,320,321,322,323,324,325,326,327,328,329,330,331,332,147,145,333,105,156,90,136,334,335,336,237,337,194,338,339,340,341,342,343,344,169,345,346,347,348,349,350,106,351,352,166,161,167,159,353,354,45,355,356,151,357,152,358,359,360,361,362,363,50,239,366,170,367,368,369,370,371,372,373,374,375,376,377,378,379,380,381,382,383,74,384,385,228,234,223,386,387,388,236,221,389,182,232,390,230,392,393,394,0,46,680,607,703,1,706,8,238,9,10,11,12,13,14,15,16 20

As mentioned, I am building a network by Web-Crawling

How Can I Add Short Arguments to a Shell Script

Posted: 05 Jul 2022 01:36 PM PDT

To add options, I have the following in one of my scripts:

parse_opts() {      while [ $# -gt 0 ]; do      case "$1" in          -h|--help)          help=1          shift          ;;          -r|--raw)          raw=1          shift          ;;          -c|--copy)          copy=1          shift          ;;          -s|--sensitive)          sensitive=1          shift          ;;          -i|--insensitive)          insensitive=1          shift          ;;          *)          shift          ;;      esac      done  }  

There is only one problem, I cannot use multiple options at the same time. For example, using -r -s works but using -rs does not. How can I programmatically make it work without adding separate entries? The shell I am using is sh.

how to open multiple new tabs in qterminal with a bash script in debian 11 lxqt

Posted: 05 Jul 2022 10:52 AM PDT

I am trying to create a bash script that setup my all development stuff in an instant.

What will script do.


  • open 5 different tabs in one window.
  1. clear terminal and start firefox-esr.
  2. clear terminal and run top command.
  3. clear terminal and start development server by using npm live-server.
  4. edit a file by using vim.
  5. for testing purpose only.

how script look like.


#!/bin/sh  qterminal --tab  #clear && top  

what error i got.


when i try to open new tab using command i got this error.

qterminal: unrecognized option '--tab'

how can i open new tabs in terminal by using commands.


If know how to make that kind of script please help me it will saves alot of my time.

VPS Port forwarding without SNAT/Masquerade using source-based routing?

Posted: 05 Jul 2022 01:37 PM PDT

I'm trying to test out using a VPS to port forward 80,443 into a local web server over a VPN tunnel while keeping the original source public IP intact. The main problem here is routing, as the edge routing device will effectively be seeing 2 interfaces with public IPs coming in (the true WAN, and the VPN interface).

Obviously, the simple solution is to just SNAT traffic going across the vpn tunnel, but I would like insight into the true source of the web traffic, plus the more NAT I do, the more overhead I introduce into the system. In addition I am avoiding putting any reverse proxy on the VPS as I refuse to decrypt the web traffic at any time while traversing the VPS.

Now on to the testing so far: IPs used: Web SRV: 192.168.0.2, VPS Wireguard Addr: 192.168.200.1, Router Wireguard Addr: 192.168.200.2.

I have a virtual test network set up with a few Debian 11 boxes serving as the VPS server, the local router, and a web server.

On the VPS, I DNAT traffic coming in on the interface on port 80,443:

iptables -t nat -A PREROUTING -i eth0 -p tcp -m multiport –dports 80,443 -j DNAT –to-destination 192.168.0.2  

I already set sysctl.conf to enable ipv4 forwarding. I have a static route on the VPS telling where to get to the 192.168.0.2:

ip route add 192.168.0.2/32 via 192.168.200.2 dev wg0  

On the local router, I found some reply based rules online based on interfaces:

echo 200 vpsrt >> /etc/iproute2/rt_tables
  ip rule add from 192.168.200.2 table vpsrt prio 1  ip route add default via 192.168.200.1 dev wg0 table vpsrt  

Since the VPN tunnel is on the router, the web server already has its default gateway configured, so return traffic between the local router and web server doesn't change at all.

wg0.conf on the VPS (acting as server):

[Interface]  PrivateKey=*  Address=192.168.200.1/30  ListenPort=50000    [Peer]  PublicKey=*  AllowedIPs=192.168.200.2/32,192.168.0.2/32  

wg0.conf on the router (acting as the client):

[Interface]  Address=192.168.200.2/32  PrivateKey=*    [Peer]  Endpoint=PUBIPVPS:50000  PublicKey=*  AllowedIPs=172.20.200.0/30  

I've verified the Wireguard tunnel works. I can ping 192.168.200.1 and .2 on each device respectively, and wg show says handshake is complete and bidirectional data is flowing.

Doing some PCAPs, traffic going through the VPS is correct. Tcpdump on wg0 on the VPS shows packets addressed to 192.168.0.2 port 443. WG Show on the VPS says its transmitting 1.3Kb, but the WG Show on the router says its only received a 500 bytes (the test ping). Tcpdump on the routers WAN interface shows the Wireguard packets are reaching the router. But the Tcpdump on the routers wg0 shows its not receiving any packets.

I am pretty sure the problem is Wireguard is silently dropping packets with a source IP that is not configured in the AllowedIPs (on the router). But the problem is I can't put 0.0.0.0/0 on routers wg0.conf because it would start using the tunnel as the default gateway for the entire network!

Is it impossible to do source based routing with Wireguard as a non-default gateway? Are there any potential work-arounds using another tunneling software? I really like the speed of Wireguard, but it appears without SNAT/Masquerade it becomes extremely limited.

Update: just found this article: https://techoverflow.net/2021/07/09/what-does-wireguard-allowedips-actually-do/ It explained that wireguards "firewalling" and routing are tied together using AllowedIPs.

A little more testing, I can simply put AllowedIPs on the router to 0.0.0.0/0 and add a postup rule in wg0.conf to remove the default gateway route and instead add a static route to the tunnel IPs only. (Unfortunate that wireguard chose to roll "firewalling" and routing into a single option)

A Tcpdump on the router's wg0 interface shows the packets are now traversing the tunnel. Tcpdump on the routers LAN interface shows the packets are transmitted out, and the web server is responding. But now the issue is those interface source-based routing rules appear to not be working as the web servers response is not being passed into the wireguard tunnel.

A Tcpdump on the routers WAN interface shows the web servers syn-acks being pushed out that gateway. So now my problem lies in the source-based routing rules I mentioned above. Source-based routing is out of my knowledge area, so any help with those is much appreciated.

Installing realtek r8168 on Fedora 36 (5.18.5)?

Posted: 05 Jul 2022 11:03 AM PDT

Linux noob here (starting out). I had a couple successful install trials before and now I am settling on with one configuration. I have now failed to get my network card to work.

Desktop PC, very old system - AMD Phenom II 955, Gigabyte GA-MA790XT-UD4P

The driver I am interested in is this: GBE Ethernet LINUX driver r8168 for kernel up to 5.17

for a background: I had used USB-Ethernet dongle to try out some of the r8168 install tutorials and dnf upgrade had updated my kernel from fresh install's 5.17 to 5.18. -> I thus suspect kernel inconsistencies but how to rework this ?

The kernel is using r8169 drivers (lspci -v), which I know doesn't work for my system! Using the driver's included autorun.sh did work for fedora 35 (I believe like half a year ago) and I have also successfully been able to install the r8168 drivers previously on Ubuntu.

Doing "$ sudo ./autorun.sh" from the folder gets me this output:

Check old driver and unload it  Build the module and install  make[2]: *** /lib/modules/5.18.5-200.fc36.x86_64/build: No such file or directory. Stop.  make[1]: *** [Makefile:158: clean] Error 2  make: *** [Makefile:48: clean] Error 2  

Thus I somehow need to force the Fedora to use r8168 instead, but I can't figure out how - WITHOUT having internet access (duh, because the GBE Ethernet card is not working). AND I want this solution to work afterwards with no USB-Ethernet dongle.

I checked the that location and it lists the build file, but opening 'build' pops-up a dialog:

The link "build" is broken.   This link cannot be used because its target "usr/src/kernels/5.18.5-200.fc36.x86_64" doesn't exist  

EDIT:

Managed to install kernel-devel for 5.18.6-200* and all dependencies. Now when running driver's autorun.sh I get this output:

Check old driver and unload it.  Build the module and install  /home/am/Documents/r8168-8.050.03/src/r8168_n.c: In function 'rtl8168_mac_loopback_test':  /home/am/Documents/r8168-8.050.03/src/r8168_n.c:3717:17: error: implicit declaration of function 'pci_dma_sync_single_for_device'; did you mean 'dma_sync_single_for_device'? [-Werror=implicit-function-declaration]   3717 |                 pci_dma_sync_single_for_device(tp->pci_dev,        |                 ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~        |                 dma_sync_single_for_device  /home/am/Documents/r8168-8.050.03/src/r8168_n.c: In function 'rtl8168_init_board':  /home/am/Documents/r8168-8.050.03/src/r8168_n.c:26448:14: error: implicit declaration of function 'pci_set_dma_mask' [-Werror=implicit-function-declaration]  26448 |             !pci_set_dma_mask(pdev, DMA_BIT_MASK(64)) &&        |              ^~~~~~~~~~~~~~~~  /home/am/Documents/r8168-8.050.03/src/r8168_n.c:26449:14: error: implicit declaration of function 'pci_set_consistent_dma_mask' [-Werror=implicit-function-declaration]  26449 |             !pci_set_consistent_dma_mask(pdev, DMA_BIT_MASK(64))) {        |              ^~~~~~~~~~~~~~~~~~~~~~~~~~~  cc1: some warnings being treated as errors  make[3]: *** [scripts/Makefile.build:288: /home/am/Documents/r8168-8.050.03/src/r8168_n.o] Error 1  make[2]: *** [Makefile:1842: /home/am/Documents/r8168-8.050.03/src] Error 2  make[1]: *** [Makefile:154: modules] Error 2  make: *** [Makefile:41: modules] Error 2  

Any great ideas to try out ?


The closest topics covering this I found were:

This doesn't state how the install was actually done, thus I have no idea how to apply this.

This ASSUMES I can access internet to do dnf upgrade's and stuff - no-go.

What is the difference between `ash` and `sh` shell on Linux?

Posted: 05 Jul 2022 11:53 AM PDT

I am new to a lot of tools on Linux. A quick search did not help, I only got more about bash or sh, and searching for it somehow leans towards bash in the search results. There are quite a few different shell types out there, I have even seen zsh, thus I guess they are all forks from sh.

There is already a question What is the difference between using bash and sh to run a script? that is similar:

  • but I could not find out what the "a" stands for in a shell
  • and this question is also not just about running a script, but the shell as such - since I can choose to take ash or sh for a docker container at hand.

I remember ash would be more comfortable though still very slim, and I have forgotten why.

What is the main difference between ash and sh? size, handling, ...?

Base image was python:3-alpine, even the one with the latest tag (2022-07) has both /bin/ash and /bin/sh. There should be a reason why the two are there in the alpine image that is made to be the most compact of all. I call it with docker-compose -f docker-compose-develop.yaml run --rm MY_CONTAINER ash.

What do these Clementine extras do? What is "Rain" and "Make it so!"?

Posted: 05 Jul 2022 11:34 AM PDT

Using the Clementine music player (1.4 rc2) on Debian 11, there is a menu named "Extras".

The two choices "Rain" and "Make it so!" can each be checked, but they don't seem to do anything (either visually, or auditorially... if that is a word now).

Can anyone provide a picture, or maybe a short explanation of what should be seen on screen when these are selected?


Yes, a quick internet search reveals that I have shamelessly copied from this question here. That is because the answer from this person says nothing. There are no words there: https://www.gitmemory.com/issue/clementine-player/Clementine/6729/653499524

mdcheck_start Service Fails to Start

Posted: 05 Jul 2022 01:40 PM PDT

I've got a brand new Ubuntu 20.04 machine built out which is using a mdadm RAID5 configuration (3x 10TB). The system is throwing an error every time I log in. I can see from systemctl that the mdcheck_start service has failed.

I can also see from checking the service status that the daemon is trying to launch a script that does not exist. This script was not installed with Ubuntu or any of the mdadm packages.

systemctl status mdcheck_start.service  ● mdcheck_start.service - MD array scrubbing       Loaded: loaded (/lib/systemd/system/mdcheck_start.service; static; vendor preset: enabled)       Active: failed (Result: exit-code) since Sun 2020-05-03 09:18:05 EDT; 5min ago  TriggeredBy: ● mdcheck_start.timer      Process: 196602 ExecStart=/usr/share/mdadm/mdcheck --duration $MDADM_CHECK_DURATION (code=exited, status=203/EXEC)     Main PID: 196602 (code=exited, status=203/EXEC)    May 03 09:18:05 BAILEYFS02 systemd[1]: Starting MD array scrubbing...  May 03 09:18:05 BAILEYFS02 systemd[196602]: mdcheck_start.service: Failed to execute command: No such file or directory  May 03 09:18:05 BAILEYFS02 systemd[196602]: mdcheck_start.service: Failed at step EXEC spawning /usr/share/mdadm/mdcheck: No such file or directory  May 03 09:18:05 BAILEYFS02 systemd[1]: mdcheck_start.service: Main process exited, code=exited, status=203/EXEC  May 03 09:18:05 BAILEYFS02 systemd[1]: mdcheck_start.service: Failed with result 'exit-code'.  May 03 09:18:05 BAILEYFS02 systemd[1]: Failed to start MD array scrubbing.  

Is this a legitimate error?

Can I safely disable this service so I stop getting these annoying errors everytime I log in? Shouldn't mdadm be scrubbing my array periodically to ensure it is working properly?

Error

Ctrl + Page UP / Page Down are reversed

Posted: 05 Jul 2022 01:24 PM PDT

I use Ubuntu, KDE and xfce. All the shortcuts for changing tabs or anything involving the keys Ctrl + Page Up instead of going right, it goes left (down) .

  1. How can I change that?
  2. Is that the correct way?

Am I the reverse here ? :)

Mutt error SMTP session failed: read error

Posted: 05 Jul 2022 01:03 PM PDT

set ssl_starttls=yes  set ssl_force_tls=yes    set imap_user = 'name@gmail.com'  set imap_pass = 'password'    set from='name@gmail.com'  set realname='name@gmail.com'    set folder = imaps://imap.gmail.com/  set spoolfile = imaps://imap.gmail.com/INBOX  set postponed="imaps://imap.gmail.com/[Gmail]/Drafts"    set header_cache = "~/.mutt/cache/headers"  set message_cachedir = "~/.mutt/cache/bodies"  set certificate_file = "~/.mutt/certificates"    set smtp_url = 'smtp://name@gmail.com:password@smtp.gmail.com:465/'    set move = no  set imap_keepalive = 900    set smtp_pass="secrets"  

When sending a mail got that error. Firewall also off on server and port 465 open on Security group. Is it required to install postfix with mutt?

Connection to smtp.gmail.com closed SMTP session failed: read error Could not send the message.  

I can read the mails on my inbox but can't send emails. This is on my maillog,

Mar  1 10:20:01 appsvr2 postfix/qmgr[6075]: 9703F3F7974: from=<root@appsvr2.localdomain>, size=768, nrcpt=1 (queue active)  Mar  1 10:20:01 appsvr2 postfix/local[5970]: 9427E3F796E: to=<root@appsvr2.localdomain>, orig_to=<root>, relay=local, delay=0.04, delays=0.03/0.01/0/0, dsn=2.0.0, status=sent (delivered to mailbox)  Mar  1 10:20:01 appsvr2 postfix/qmgr[6075]: 9427E3F796E: removed  Mar  1 10:20:01 appsvr2 postfix/local[5970]: 9703F3F7974: to=<root@appsvr2.localdomain>, orig_to=<root>, relay=local, delay=0.04, delays=0.03/0.01/0/0, dsn=2.0.0, status=sent (delivered to mailbox)  Mar  1 10:20:01 appsvr2 postfix/qmgr[6075]: 9703F3F7974: removed  Mar  1 10:30:01 appsvr2 postfix/pickup[5945]: A43FB3E643: uid=0 from=<root>  Mar  1 10:30:01 appsvr2 postfix/cleanup[5980]: A43FB3E643: message-id=<20190301050001.A43FB3E643@appsvr2.localdomain>  Mar  1 10:30:01 appsvr2 postfix/qmgr[6075]: A43FB3E643: from=<root@appsvr2.localdomain>, size=827, nrcpt=1 (queue active)  Mar  1 10:30:01 appsvr2 postfix/local[5983]: A43FB3E643: to=<root@appsvr2.localdomain>, orig_to=<root>, relay=local, delay=0.04, delays=0.03/0.01/0/0, dsn=2.0.0, status=sent (delivered to mailbox)  Mar  1 10:30:01 appsvr2 postfix/qmgr[6075]: A43FB3E643: removed  

set php-fpm user and group to website's owner

Posted: 05 Jul 2022 11:42 AM PDT

I have a centos 6 server which using apache to connect php-fpm to run php site, i use user's home directory for the website root, for example /home/user1, . On the other hand, the php-fpm www.conf, the lines user and group i set to apache.

So, when the php write or create a file (e.g /home/user1/var/log/file), it will use apache:apache as owner. Then i need to run a command at /home/user1/var/log/ "chmod -R ug+s ." to make php to create a file as apache:user1 while user1 is group

So my question is, it is possible to set php-fpm to create file using the directory user owner (user1:user1) but not apache:apache

DEGRADED zpool can't be imported with I/O error

Posted: 05 Jul 2022 12:02 PM PDT

I'm running a personal server at home with a CentOS 7 OS and a 12TB zpool. It's been running for a couple of years and yesterday I noticed some problems so I went in to have a look.

At first it seemed like one of my drives had failed, with zpool import giving the following results:

pool: media  id: 1363376331138686016  state: DEGRADED  status: One or more devices contains corrupted data.  action: The pool can be imported despite missing or damaged devices.    The  fault tolerance of the pool may be compromised if imported.     see: http://zfsonlinux.org/msg/ZFS-8000-4J   config:    media                                DEGRADED    raidz1-0                           DEGRADED      ata-ST3000DM001-1ER166_W500G55Q  ONLINE      ata-ST3000DM001-1CH166_Z1F278KB  UNAVAIL      sdc                              ONLINE      sde                              ONLINE      sdf                              ONLINE  

This looks ok, however I can't seem to import the pool directly.

Running zpool import media gives me:

cannot import 'media': I/O error  Destroy and re-create the pool from  a backup source.  

I've looked around to figure out the problem but everything I've found has given me nothing.

Some other things I've tried:

  • zpool import -fFX: zpool could not be imported
  • zpool import -fFV: zpool imported with FAULTED status
  • zpool status: no pools available

Is anyone able to point me in the right direction? I'm not sure what my next course of diagnosis should be.

Zsh tab completions not working as desired for partial paths

Posted: 05 Jul 2022 11:08 AM PDT

I want case-insensitive fuzzy completion for files and directories in zsh. After reading the manual for a few hours, this is what I came up with:

zstyle ':completion:*:*:*:*:globbed-files' matcher 'r:|?=** m:{a-z\-}={A-Z\_}'  zstyle ':completion:*:*:*:*:local-directories' matcher 'r:|?=** m:{a-z\-}={A-Z\_}'  zstyle ':completion:*:*:*:*:directories' matcher 'r:|?=** m:{a-z\-}={A-Z\_}'  

Additionally, I want pressing TAB once to display possible completions, only modifying what I have typed if there is exactly one completion. Then pressing TAB a second time should put me into "menu completion" mode. Based on the manuals, I came up with this:

zstyle ':completion:*' menu select  

Now everything works as it should except in one circumstance. I have two folders Desktop and .rstudio-desktop in my home directory. Since I have setopt globdots, I expect typing the following:

$ cd ~/dktop<TAB>  

to leave my command as entered, and display as completion candidates Desktop and .rstudio-desktop. Instead, it removes dktop, leaving me with the following:

$ cd ~/  

I have looked at all of the relevant manuals, guides, Stack Exchange questions, and various other sources. But whatever I do, I can't make this work.

Interestingly, though, if I'm in the home directory and type the following then everything works as expected:

$ cd dktop<TAB>  

That is, it's only a problem with non-leading segments of paths (and you can see with C-x h that this corresponds to the directories tag rather than the local-directories tag being used).

For easy reproducibility, here is a ~/.zshrc that will reproduce the situation and behavior described above (tested on a fresh El Capitan virtual machine with zsh from Homebrew).

Setting up DD-WRT behind a DSL router/modem from ISP

Posted: 05 Jul 2022 02:01 PM PDT

Okay so I bought a Buffalo WBMR-HP-G300H the first router/modem that supports DD-WRT, after having managed to get dd-wrt to run on the router without problems the next step was to plug my dsl cable from my isp into it.

The supported adsl standards of my ISP are: ITU G.992.1 (ADSL-G.DMT) ITU G.992.2 (ADSL-G.Lite) ITU G.992.3 (ADSL2) ITU G.992.5 (ADSL2+)

But when I access my DD-WRT setup page I can only see Annex A Annex A T1 Annex A Lite Annex A DMT Annex A ADSL2 Annex A ADSL2+ Annex L Annex B Annex B DMT Annex B ADSL2 Annex B ADSL2+ Annex M Annex M ADSL2 Annex M ADSL2+

After consulting https://en.wikipedia.org/wiki/Asymmetric_digital_subscriber_line#ADSL_standards

Not much has remained clear in terms of which one should I choose or are even the ones I need available. My connection speed is 80/10mbit/s

As this is the configuration on my ISP router/modem Encapsulation: 1483 Bridged IP LLC VPI: 1 VCI: 64 Bridge Mode: Enabled

But regardless the dd-wrt router just does not work, the dsl just blinks periodically and that is it.

Now the second thing that I have attempted to do is to just connect my dd-wrt rotuer to my ISP router but the two just do not play well with each other at all.

I select the Automatic Configuration - DCHP I ve set the router IP ( dd-wrt ) to be 192.168.1.50 just to test things out, while the IP of the ISP router is 192.168.1.1

One strange thing to me is that when I select Automatic Configuration DHCP is that it is still asking me for the Annex Type VPI/VCI etc... even though I am just configuring the LAN ports on both ends.

I am sorry if my description is not the best I really am, if you require additional information please ask I will provide it promptly.

Thank you all for your time

Why does Vim indent pasted code incorrectly?

Posted: 05 Jul 2022 12:07 PM PDT

In Vim, if I paste this script:

#!/bin/sh  VAR=1  while ((VAR <  10))      do          echo "VAR1 is now $VAR"          ((VAR = VAR +2))      done      echo "finish"  

I get these strange results:

#!/bin/sh  #VAR=1  #while ((VAR <  10))  #       do  #                       echo "VAR1 is now $VAR"  #                                       ((VAR = VAR +2))  #                                               done  #                                                       echo "finish"  #                                                         

Hash signs (#) and tabs have appeared. Why?

How do I close a terminal without saving the history?

Posted: 05 Jul 2022 12:18 PM PDT

More than once I've accidentally run a number of commands and polluted my bash history. How do I close my terminal without saving my bash history? I'm using Fedora.

No comments:

Post a Comment