Difference between revisions of "Git"

From ITSwiki
Jump to: navigation, search
[unchecked revision][unchecked revision]
(moving existing repos into gitolite)
(Instructions for collaboration with user from outside DTU)
 
(86 intermediate revisions by 3 users not shown)
Line 1: Line 1:
= How to get an account =
+
To use git for your own and team projects, DTU Compute is providing a GitLab instance with all the benefits it offers. Primarily a place for your repositories, other features are or will become available.
  
IMM employees and Ph.D's can get a git account by sending an email to [mailto:ITservice@imm.dtu.dk IT Service] with  your SSH public key which you find in <code>~/.ssh/*pub</code>. E.g. <code>~/.ssh/id_rsa.pub</code>.
 
  
If you don't have a public key then you can create one by
+
Login with your DTU credentials here: https://lab.compute.dtu.dk
<pre>
+
ssh-keygen -N '' -t rsa -f ~/.ssh/id_rsa
+
</pre>
+
  
It is important that you keep your private key (<code>~/.ssh/id_rsa</code>) and public key safe and have a backup. Should someone get hold of your private key, then all your git repositories can get accessed by that person.
 
 
== Disk space ==
 
 
By default the git admin get 1GB of disk space and a hard limit of 1,5GB, where all the git admin's repositories combined can not exceed these limits.
 
 
This means that if you have created e.g. 3 repos and have added some users to these, and if just one of these users adds a big file by mistake, and exceed the 1GB limit, then no one will be allowed to push to any of the git admin's repos.
 
 
If the disk usage is e.g. 900MB, and a user pushes a commit of 700MB, then the file system will  stop writing when 1,5GB is reached, and there is a potential data loss for the repos owned by the git admin.
 
 
Linus is not shy to say that Git is not designed to contain big binary blobs. [http://osdir.com/ml/git/2009-05/msg00051.html Example].
 
 
So 1GB is an enormous amount of space, if your uses use it as intended, so these limits should never be a problem. The limits are there however to protect the other git admin's repos, so a user from gitAdmin1 can not impact other admins' repos.
 
 
If you have a need for version control on big amounts binary blobs of data (e.g. images), then please contact [mailto:ITservice@imm.dtu.dk IT Service] and will try to find a solution for this.
 
 
=== Disk usage output ===
 
 
Everytime a <code>git push</code> is made by any of the git admin's users the total amount of used disk space is output.
 
 
Example:
 
  
 +
== Add your public keys ==
 +
To access your private repositories, add your public SSH-key to your GitLab account.<br/>
 +
If you don't have a SSH-key, on Linux/MacOS you can use this command in your favourite shell:
 +
<pre style="background-color: #374048; color: white; border-style: none; padding: 5px; width: 75%;">
 +
ssh-keygen
 +
</pre>
 +
Windows 10 has a native SSH client; in PowerShell create a key pair with:
 +
<pre style="background-color: #374048; color: white; border-style: none; padding: 5px; width: 75%;">
 +
ssh-keygen.exe
 +
</pre>
 +
Press <code>Enter</code> 3 times to create a password-less key pair, located in:
 
<pre>
 
<pre>
mttj@pcmttj:~/git-test/test$ touch 7 && git add 7 && git commit -a -m "test" && git push
+
$HOME/.ssh/id_rsa
[master 25b388e] test
+
$HOME/.ssh/id_rsa.pub
0 files changed
+
create mode 100644 7
+
Counting objects: 3, done.
+
Delta compression using up to 2 threads.
+
Compressing objects: 100% (2/2), done.
+
Writing objects: 100% (2/2), 220 bytes, done.
+
Total 2 (delta 1), reused 0 (delta 0)
+
remote:                                                                  <-------------
+
remote: user4@git.imm.dtu.dk have used 3 MB out of 1000 MB.              <-------------
+
remote:                                                                  <-------------
+
To user4@git.imm.dtu.dk:test.git
+
  78b3caf..25b388e  master -> master
+
mttj@pcmttj:~/git-test/test$
+
 
</pre>
 
</pre>
 +
The file <code>id_rsa.pub</code> is your public key that you can share with other systems, including DTU Compute's GitLab.
  
If a user have made a big commit that have exceeded the quota limit then no one can push to any of the git admin's repo's.
 
  
Example:
+
To add your public key in GitLab, follow these steps after login:
<pre>
+
mttj@pcmttj:~/git-test/test$ touch 8 && git add 8 && git commit -a -m "test" && git push
+
[master e223957] test
+
0 files changed
+
create mode 100644 8
+
Counting objects: 3, done.
+
Delta compression using up to 2 threads.
+
Compressing objects: 100% (2/2), done.
+
Writing objects: 100% (2/2), 228 bytes, done.
+
Total 2 (delta 1), reused 0 (delta 0)
+
remote:
+
remote: user4@git.imm.dtu.dk have used 3 MB and exceeded quota of 1 MB.  <-------------
+
remote: No changes have been made to remote repository.                  <-------------
+
remote:
+
To user4@git.imm.dtu.dk:testgit
+
! [remote rejected] master -> master (pre-receive hook declined)        <-------------
+
error: failed to push some refs to 'user4@git.imm.dtu.dk:test.git'        <-------------
+
mttj@pcmttj:~/git-test/test$
+
</pre>
+
  
== Deleting a big commit ==
 
  
TODO
+
1) In the top-right corner click your user icon<br/>
 +
2) Click '''Settings''' and on the left-hand panel click '''SSH Keys '''<br/>
 +
3) Follow the instructions from here<br/>
  
== Rename or delete repo ==
 
  
TODO
+
== Give access to other users ==
 +
To share a repository with other users, the user you want to add has to login to GitLab first. Then you can find and add them as a member of your project.
  
= Creating repositories and setting permissions =
 
  
Once your account have been created, you can clone the <code>gitolite-admin</code> repo by
+
1) Select your project you want to share<br/>
<pre>
+
2) In the left-hand panel select '''Settings -> Members'''<br/>
git clone dtuInitials@git.imm.dtu.dk:gitolite-admin
+
3) Under '''Select members to invite''' search for the username<br/>
</pre>
+
where <code>dtuInitials</code> should be replaced by your DTU initials which are 4 or more letters long.
+
  
This repository is where you create repositories, add users, and set permissions for the users.
 
  
The procedure to create a repo and add a user is:
+
=== Instructions for collaboration with user from outside DTU ===
 +
Users can get access to Gitlab service at DTU Compute only if they have a DTU account. Therefore, if you’d try giving access to an external person through your Gitlab project/group settings, it will not work.
  
* Specify the repo in <code>gitolite-admin/conf/gitolite.conf</code>
 
* Specify which users should be allowed and with what permissions
 
* Rename the users public key
 
* Add the users key to <code>gitolite-admin/keydir/</code>
 
* git add, commit, push <code>gitolite-admin</code>
 
  
All this will be explained in details in the coming sections.
+
1) Create a guest account for the external person by logging in to https://guest.dtu.dk with your DTU user and password.<br/>
 +
2) After login in, you should see the options as in the figure below. Choose Create user.<br/>
 +
<br/>[[File:Guestdtudk.png|800px]]<br/><br/>
 +
3) You should see the form below appearing in the next screen. The username must consist of 8-9 symbols (only letters and digits are allowed). It is important to enable access to DTU Wireless system for Gitlab to be able to discover the new user.<br/>
 +
<br/>[[File:Createguestuser.png|800px]]<br/><br/>
 +
4) After clicking Create user button, you should see the green success message and the information about the user appearing below the search box. You can edit, delete, and change the password for users. Note, that guest users are not able to change credentials themselves and you must do it for them.<br/>
 +
5) After certain amount of time (around 1-2 hours), notify the owners of the guest accounts that they should be able to sign in to Gitlab with their new credentials at the Gitlab logon page shown below. The credentials can be sent by email while changing user’s password.<br/>
 +
<br/>[[File:Gitlab login guests.png|800px]]<br/><br/>
 +
6) You should now be able to share your group/project with your guest users by going to Members setting and clicking the blue Invite members button. The search box will appear where you can search guest users by their usernames in the usual way.<br/>
  
== Example ==
+
== Project Visibility & License ==
 +
New repositories are per default set to be '''private''' which means that only you, and members that you allow, are able to see and work with the repository. This can be changed to either '''internal''' (lab.compute.dtu.dk members) or '''public''' (visible to the whole internet). To change the visibility of a repository:
  
The git admin is the only one by default who can pull the gitolite-admin repo.
 
  
To add a repo edit <code>gitolite-admin/conf/gitolite.conf</code>.
+
1) Select '''Settings -> General'''<br/>
 +
2) Expand '''Visibility, project features, permissions''' and change '''Project visibility'''
  
Example:
 
<pre>
 
@staff            =  dilbert alice
 
@projects          =  mobile fast_algo
 
  
repo @projects inv_prob
+
'''NOTE:''' Please be aware that public repositories can be found in the <code>/public</code> access directory (https://lab.compute.dtu.dk/public). If you have projects here that are '''not''' meant to be public, change the visibilty of your repository!
  RW+            =  @staff
+
  -      master  =  ashok
+
  RW              =  ashok
+
  R              =  wally
+
</pre>
+
  
Aside from this the public keys from dilbert, alice, ashok, and wally have to be added, and finally everything committed and pushed.
+
If you do want your project to be public, we ask that you add a license. This can be done easily by clicking the '''Add license''' link beneath the project name in your project homepage.
  
== Syntax ==
 
  
In general, everything is space separated; there are no commas, semicolons, etc., in the syntax. Comments starts with <code>#</code>.
+
== Mirroring repositories ==
 +
It may make sense to mirror a repository to/from an external source such as Github. For instructions read:<br/>
 +
https://lab.compute.dtu.dk/help/user/project/repository/repository_mirroring.md
  
User names and repo names must start with an alphanumeric, but after that they can also contain <code>.</code>, <code>_</code>, or <code>-</code>.
 
  
Usernames can optionally be followed by an <code>@</code> and a domainname containing at least one <code>.</code> (this allows you to use an email address as someone's username). Repo names can contain <code>/</code> characters (this allows you to put your repos in a tree-structure for convenience).
+
= Migrating from Gitolite to GitLab =
 +
Our Gitolite server (git.compute.dtu.dk) is considered deprecated, and everyone with active repositories there are encouraged to migrate. This can be done from a local PC with a cloned repo.
  
There are no continuation lines by default. You do not need them; the [[#Groups|Groups]] section will tell you how you can break up large lists of names in a group definition into multiple lines.
+
Migrating a repo requires a bit of manual work, but not much. What it basically comes down to, is moving the ''origin'' from Gitolite to GitLab. Here are the steps described as simple as possible:
  
== Groups ==
+
* Make sure your locally cloned repo - incl. all branches, tags, etc. - is up to date with ''origin''
 +
* Create a new repository on GitLab using the same name as the repo you want to move
 +
* Add a new ''remote'' using the URL from GitLab, and push everything to this ''remote''
 +
* Remove all references to the old ''remote'' (Gitolite)
 +
* Rename the new ''remote'' to ''origin''
 +
* In GitLab add the users you want to share the project with.
 +
* Users may want to clone the repo anew rather than change ''origin'' manually.
  
You can group repos or users for convenience. The syntax is the same for both and does not distinguish; until you use the group name it could really be either.
+
== Steps described ==
 +
As is shown above, it is pretty straight forward to move a repository, but as always the details are important. Here the steps will be described a bit more together with the commands to use.
  
Here's an example:
+
First make sure that your local repository clone is up to date with ''origin'' (the gitolite server), and make sure that you checkout all remote branches locally.
<pre>
+
<pre style="background-color: #374048; color: white; border-style: none; padding: 5px; width: 75%;">
@developers    =  dilbert alice wally
+
git fetch origin
 +
git branch -a
 +
git checkout -b feature1 origin/feature1
 
</pre>
 
</pre>
Group definitions accumulate; this is the same as the above:
 
<pre>
 
@developers    =  dilbert
 
@developers    =  alice
 
@developers    =  wally
 
</pre>
 
You can use one group in another group definition; the values will be expanded right there (meaning later additions will not appear in the second group).
 
<pre>
 
@developers    =  dilbert alice
 
@interns        =  ashok
 
@staff          =  @interns @developers
 
@developers    =  wally
 
  
# wally is NOT part of @staff
+
Create the new replacement repository at https://lab.compute.dtu.dk and copy the SSH clone URL - eg. <code>git@lab.compute.dtu.dk:user/project2.git</code> - to add a new ''remote'' in your local repo, and push all branches to this new ''remote'', here called "new-origin".
 +
<pre style="background-color: #374048; color: white; border-style: none; padding: 5px; width: 75%;">
 +
git remote add new-origin git@lab.compute.dtu.dk:user/project2.git
 +
git push --all new-origin
 +
git push --tags new-origin # if you use tags
 
</pre>
 
</pre>
  
<code>@all</code> is a special group name that is often convenient to use if you really mean "all repos" or "all users".
+
Delete the reference to the old ''origin'' and make the new remote the origin.
 
+
<pre style="background-color: #374048; color: white; border-style: none; padding: 5px; width: 75%;">
== Adding and removing users ==
+
git remote rm origin
 
+
git remote rename new-origin origin
To add a user, e.g. Alice, obtain her public key (typically <code>$HOME/.ssh/id_rsa.pub</code> and then
+
 
+
mv id_rsa.pub gitolite-admin/keydir/alice.pub
+
cd gitolite-admin
+
git add keydir/alice.pub
+
git commit -a -m "Added alice.pub"
+
git push
+
 
+
The name of the key (alice in this case) will be the name you should use in <code>gitolite-admin/conf/gitolite.conf</code>.
+
 
+
To remove a user
+
 
+
cd gitolite-admin
+
git rm keydir/alice.pub
+
git commit -a -m "Removed alice.pub"
+
git push
+
 
+
== adding and removing repos ==
+
 
+
NOTE: this page describes how to add new repos. To bring already existing repos into gitolite control, click here.
+
 
+
WARNING: Do NOT add new repos or users manually on the server. Gitolite users, repos, and access rules are maintained by making changes to a special repo called 'gitolite-admin' and pushing those changes to the server.
+
 
+
Just as for users, all operations are in a clone of the gitolite-admin repo.
+
 
+
To add a new repo, edit conf/gitolite.conf and add it, along with at least one user with some permissions. Or add it to an existing repo line:
+
 
+
repo gitolite tsh gitpod
+
    RW+    =  sitaram
+
    RW  dev =  alice bob
+
    R      =  @all
+
The "repo" line can have any number of repo names or repo group names in it. However, it can only be one line; this will not work
+
 
+
repo foo
+
repo bar    # WRONG; 'foo' is now forgotten
+
    RW      =  alice
+
If you have too many, use a group name:
+
 
+
@myrepos    =  foo
+
@myrepos    =  bar
+
 
+
repo @myrepos
+
    RW      =  alice
+
Finally, you add, commit, and push this change. Gitolite will create a bare, empty, repo on the server that is ready to be cloned.
+
 
+
Removing a repo is not so straightforward. You certainly must remove the appropriate lines from the conf/gitolite.conf file, but gitolite will not automatically delete the repo from the server. You have to log on to the server and do the dirty deed yourself :-)
+
 
+
It is best to make the change in the conf file, push it, and then go to the server and do what you need to.
+
 
+
Renaming a repo is also not automatic. Here's what you do (and the order is important):
+
 
+
Go to the server and rename the repo at the Unix command line.
+
Change the name in the conf/gitolite.conf file and add/commit/push.
+
 
+
== Moving existing repos into git.imm.dtu.dk ==
+
 
+
Needs to be done on server side. Contact ITservice.
+
 
+
== Accees rules ==
+
 
+
This is arguably the most complex part of day-to-day gitolite. There are several interconnected ideas that make this hard to lay out easily if you're totally new to this, so read carefully.
+
 
+
We will use this as a running example:
+
 
+
@staff          =  dilbert alice wally bob
+
 
+
repo foo
+
    RW+        =  dilbert    # line 1
+
    RW+ dev    =  alice      # line 2
+
    -          =  wally      # line 3
+
    RW  temp/  =  @staff      # line 4
+
    R          =  ashok      # line 5
+
what does a rule look like?
+
 
+
A rule line has the structure
+
 
+
<permission> <zero or more refexes> = <one or more users/user groups>
+
The most common permissions used are:
+
 
+
R, for read only
+
RW, for push existing ref or create new ref
+
RW+, for "push -f" or ref deletion allowed (i.e., destroy information)
+
- (the minus sign), to deny access.
+
 
+
=== Less commonly used types of permissions ===
+
 
+
Git supplies enough information to the update hook to be able to distinguish several types of writes.
+
 
+
The most common are:
+
 
+
RW -- create a ref or fast-forward push a ref. No rewinds or deletes.
+
RW+ -- create, fast-forward push, rewind push, or delete a ref.
+
Sometimes you want to allow people to push, but not create a ref. Or rewind, but not delete a ref. The C and D qualifiers help here.
+
 
+
When a rule specifies RWC or RW+C, then rules that do NOT have the C qualifier will no longer permit creating a ref.
+
 
+
Please do not confuse this with the standalone C permission that allows someone to create a repo
+
 
+
When a rule specifies RWD or RW+D, then rules that do NOT have the D qualifier will no longer permit deleting a ref.
+
 
+
Note: These two can be combined, so you can have RWCD and RW+CD as well.
+
 
+
One very rare need is to reject merge commits (a commit series that is not a straight line of commits). The M qualifier helps here:
+
 
+
When a rule has M appended to the permissions, rules that do NOT have it will reject a commit sequence that contains a merge commit (i.e., they only accept a straight line series of commits).
+
 
+
== Activate the new repo and user ==
+
 
+
 
+
 
+
 
+
=== ? ===
+
 
+
<pre>
+
git clone git-owner@git.imm.dtu.dk:gitolite-admin
+
 
+
scp mttj@pcmttj:.ssh/*pub    gitolite-admin/keydir/mttj.pub
+
sed -i 's/mttj@pcmttj/mttj/g' gitolite-admin/keydir/mttj.pub
+
 
+
cd gitolite-admin
+
git add keydir/mttj.pub
+
vi conf/gitolite.conf
+
git commit -a -m "Added mttj pub key and RW perm to mttj on fast_algo repo"
+
git push
+
 
</pre>
 
</pre>
  
mttj can now access fast_algo repo by
+
Your repository is now migrated to GitLab.
  
git clone git-owner@git.imm.dtu.dk:fast_algo
 
  
= Making the first git push to a new repo =
+
= More to come =
 
+
<pre>
+
git push origin master
+
</pre>
+
  
= Warnings and errors =
+
...
  
* x11 forward
+
[[Category:IT]]
* space
+

Latest revision as of 16:44, 13 November 2023

To use git for your own and team projects, DTU Compute is providing a GitLab instance with all the benefits it offers. Primarily a place for your repositories, other features are or will become available.


Login with your DTU credentials here: https://lab.compute.dtu.dk


Add your public keys

To access your private repositories, add your public SSH-key to your GitLab account.
If you don't have a SSH-key, on Linux/MacOS you can use this command in your favourite shell:

ssh-keygen

Windows 10 has a native SSH client; in PowerShell create a key pair with:

ssh-keygen.exe

Press Enter 3 times to create a password-less key pair, located in:

$HOME/.ssh/id_rsa
$HOME/.ssh/id_rsa.pub

The file id_rsa.pub is your public key that you can share with other systems, including DTU Compute's GitLab.


To add your public key in GitLab, follow these steps after login:


1) In the top-right corner click your user icon
2) Click Settings and on the left-hand panel click SSH Keys
3) Follow the instructions from here


Give access to other users

To share a repository with other users, the user you want to add has to login to GitLab first. Then you can find and add them as a member of your project.


1) Select your project you want to share
2) In the left-hand panel select Settings -> Members
3) Under Select members to invite search for the username


Instructions for collaboration with user from outside DTU

Users can get access to Gitlab service at DTU Compute only if they have a DTU account. Therefore, if you’d try giving access to an external person through your Gitlab project/group settings, it will not work.


1) Create a guest account for the external person by logging in to https://guest.dtu.dk with your DTU user and password.
2) After login in, you should see the options as in the figure below. Choose Create user.

Guestdtudk.png

3) You should see the form below appearing in the next screen. The username must consist of 8-9 symbols (only letters and digits are allowed). It is important to enable access to DTU Wireless system for Gitlab to be able to discover the new user.

Createguestuser.png

4) After clicking Create user button, you should see the green success message and the information about the user appearing below the search box. You can edit, delete, and change the password for users. Note, that guest users are not able to change credentials themselves and you must do it for them.
5) After certain amount of time (around 1-2 hours), notify the owners of the guest accounts that they should be able to sign in to Gitlab with their new credentials at the Gitlab logon page shown below. The credentials can be sent by email while changing user’s password.

Gitlab login guests.png

6) You should now be able to share your group/project with your guest users by going to Members setting and clicking the blue Invite members button. The search box will appear where you can search guest users by their usernames in the usual way.

Project Visibility & License

New repositories are per default set to be private which means that only you, and members that you allow, are able to see and work with the repository. This can be changed to either internal (lab.compute.dtu.dk members) or public (visible to the whole internet). To change the visibility of a repository:


1) Select Settings -> General
2) Expand Visibility, project features, permissions and change Project visibility


NOTE: Please be aware that public repositories can be found in the /public access directory (https://lab.compute.dtu.dk/public). If you have projects here that are not meant to be public, change the visibilty of your repository!

If you do want your project to be public, we ask that you add a license. This can be done easily by clicking the Add license link beneath the project name in your project homepage.


Mirroring repositories

It may make sense to mirror a repository to/from an external source such as Github. For instructions read:
https://lab.compute.dtu.dk/help/user/project/repository/repository_mirroring.md


Migrating from Gitolite to GitLab

Our Gitolite server (git.compute.dtu.dk) is considered deprecated, and everyone with active repositories there are encouraged to migrate. This can be done from a local PC with a cloned repo.

Migrating a repo requires a bit of manual work, but not much. What it basically comes down to, is moving the origin from Gitolite to GitLab. Here are the steps described as simple as possible:

  • Make sure your locally cloned repo - incl. all branches, tags, etc. - is up to date with origin
  • Create a new repository on GitLab using the same name as the repo you want to move
  • Add a new remote using the URL from GitLab, and push everything to this remote
  • Remove all references to the old remote (Gitolite)
  • Rename the new remote to origin
  • In GitLab add the users you want to share the project with.
  • Users may want to clone the repo anew rather than change origin manually.

Steps described

As is shown above, it is pretty straight forward to move a repository, but as always the details are important. Here the steps will be described a bit more together with the commands to use.

First make sure that your local repository clone is up to date with origin (the gitolite server), and make sure that you checkout all remote branches locally.

git fetch origin
git branch -a
git checkout -b feature1 origin/feature1

Create the new replacement repository at https://lab.compute.dtu.dk and copy the SSH clone URL - eg. git@lab.compute.dtu.dk:user/project2.git - to add a new remote in your local repo, and push all branches to this new remote, here called "new-origin".

git remote add new-origin git@lab.compute.dtu.dk:user/project2.git
git push --all new-origin
git push --tags new-origin # if you use tags

Delete the reference to the old origin and make the new remote the origin.

git remote rm origin
git remote rename new-origin origin

Your repository is now migrated to GitLab.


More to come

...