Add Multi-Scanner test file

1. Add Multi-Scanner test file for harbor is deployed with multi scanners;
2. Modify notary doc description;
3. Add paragraph for e2e-api-python-based-scripting-guide;
4. Fix delete project issue;
5. Remove count qoutas in nightly and modify some of it;
6. Add Trivy in git hub offline action;

Signed-off-by: danfengliu <danfengl@vmware.com>

docs/1.10/build-customize-contribute/e2e-api-python-based-scripting-guide.md

@@ -3,7 +3,7 @@ title: E2E (API) Python Based Test Scripting Guide
 draft: true
 ---
 
-**Preparation**
+#### Preparation ####
 
 After getting Harbor source code (git clone https://github.com/goharbor/harbor.git), Harbor E2E API  test scripts  can be found in tests/apitests/python directory.
 
@@ -30,7 +30,7 @@ Harbor E2E API test scripts will import python library under archive "harborclie
 
 Until now, we have all preparation work done.
 
-**Scripting**
+#### Scripting ####
 
 As you can see, we will use python library made by "make swagger_client", in this library, we have all API functions and models, but for more convenience we encapsulate one more level in archive "library", so the script structure is as bellow:
 
@@ -47,8 +47,8 @@ As you can see, we will use python library made by "make swagger_client", in thi
 You can add both library code and script code, since not all APIs have been encapsulated.
 
 
-**Execution Example**
+#### Manual Execution Example ####
-```
+
 root@harbor:/harbor/code/harbor# `python ./tests/apitests/python/test_add_sys_label_to_tag.py`
 
 2020-03-11 13:40:07,269 DEBUG Starting new HTTPS connection (1): 1.1.1.1:443
@@ -67,10 +67,15 @@ header: Content-Security-Policy: frame-ancestors 'none'
 2020-03-11 13:40:07,482 DEBUG https://1.1.1.1:443 "POST /api/v2.0/users HTTP/1.1" 201 0
 2020-03-11 13:40:07,483 DEBUG response body:
 
-...
+......
+
+......
+
+......
+
+#### How To Trigger Script In CI ####
+
+If you like to have your scripts running in CI which is for verification of pull requests, please add your scripts into *https://github.com/goharbor/harbor/blob/master/tests/robot-cases/Group0-BAT/API_DB.robot* file, then scripts can be triggered once there is a pull request.
 
-...
 
-...
 
-```

docs/1.10/working-with-projects/project-configuration/implementing-content-trust.md

@@ -19,7 +19,7 @@ The root key is generated at: `/root/.docker/trust/private/root_keys`
 You will also be asked to enter a new passphrase for the image. This is generated at `/root/.docker/trust/private/tuf_keys/[registry name] /[imagepath]`.
 If you are using a self-signed cert, make sure to copy the CA cert into `/etc/docker/certs.d/10.117.169.182` and `$HOME/.docker/tls/10.117.169.182:4443/`. When an image is signed, it is indicated in the Web UI.
 
-When an image is signed, it has a tick shown in UI; otherwise, a cross sign(X) is displayed instead.
+A signed image will have a checkbox next to it, otherwise an X is displayed instead.
 
 If you want to remove a tag signature from harbor, you can use 'notary remove' command:
 

tests/ci/distro_installer.sh

@@ -3,5 +3,5 @@ set -x
 
 set -e
 
-sudo make package_online VERSIONTAG=dev-travis PKGVERSIONTAG=dev-travis UIVERSIONTAG=dev-travis GOBUILDIMAGE=golang:1.13.8 COMPILETAG=compile_golangimage NOTARYFLAG=true CLAIRFLAG=true CHARTFLAG=true HTTPPROXY=
+sudo make package_online VERSIONTAG=dev-travis PKGVERSIONTAG=dev-travis UIVERSIONTAG=dev-travis GOBUILDIMAGE=golang:1.13.8 COMPILETAG=compile_golangimage NOTARYFLAG=true CLAIRFLAG=true CHARTFLAG=true TRIVYFLAG=true HTTPPROXY=
-sudo make package_offline VERSIONTAG=dev-travis PKGVERSIONTAG=dev-travis UIVERSIONTAG=dev-travis GOBUILDIMAGE=golang:1.13.8 COMPILETAG=compile_golangimage NOTARYFLAG=true CLAIRFLAG=true CHARTFLAG=true HTTPPROXY=
+sudo make package_offline VERSIONTAG=dev-travis PKGVERSIONTAG=dev-travis UIVERSIONTAG=dev-travis GOBUILDIMAGE=golang:1.13.8 COMPILETAG=compile_golangimage NOTARYFLAG=true CLAIRFLAG=true CHARTFLAG=true TRIVYFLAG=true HTTPPROXY=

tests/resources/Harbor-Pages/Configuration.robot

@@ -334,5 +334,5 @@ Get Project Count Quota Text From Project Quotas List
 Get Project Storage Quota Text From Project Quotas List
     [Arguments]    ${project_name}
     Switch To Project Quotas
-    ${storage_quota}=    Get Text    xpath=//project-quotas//clr-datagrid//clr-dg-row[contains(.,'${project_name}')]//clr-dg-cell[4]//label
+    ${storage_quota}=    Get Text    xpath=//project-quotas//clr-datagrid//clr-dg-row[contains(.,'${project_name}')]//clr-dg-cell[3]//label
     [Return]  ${storage_quota}

tests/resources/TestCaseBody.robot

@@ -133,13 +133,14 @@ Body Of Admin Push Signed Image
 Delete A Project Without Sign In Harbor
     [Arguments]  ${harbor_ip}=${ip}  ${username}=${HARBOR_ADMIN}  ${password}=${HARBOR_PASSWORD}
     ${d}=    Get Current Date    result_format=%m%s
-    Create An New Project  project${d}
+    ${project_name}=  Set Variable  000${d}
-    Push Image  ${harbor_ip}  ${username}  ${password}  project${d}  hello-world
+    Create An New Project  ${project_name}
-    Project Should Not Be Deleted  project${d}
+    Push Image  ${harbor_ip}  ${username}  ${password}  ${project_name}  hello-world
-    Go Into Project  project${d}
+    Project Should Not Be Deleted  ${project_name}
-    Delete Repo  project${d}
+    Go Into Project  ${project_name}
+    Delete Repo  ${project_name}
     Navigate To Projects
-    Project Should Be Deleted  project${d}
+    Project Should Be Deleted  ${project_name}
 
 Manage Project Member Without Sign In Harbor
     [Arguments]  ${sign_in_user}  ${sign_in_pwd}  ${test_user1}=user005  ${test_user2}=user006  ${is_oidc_mode}=${false}

tests/resources/Util.robot

@@ -228,7 +228,7 @@ Command Should be Failed
 
 Retry Keyword When Error
     [Arguments]  ${keyword}  @{elements}
-    :For  ${n}  IN RANGE  1  6
+    :For  ${n}  IN RANGE  1  3
     \    Log To Console  Trying ${keyword} elements @{elements} ${n} times ...
     \    ${out}  Run Keyword And Ignore Error  ${keyword}  @{elements}
     \    Log To Console  Return value is ${out} and ${out[0]}
@@ -253,7 +253,7 @@ Retry Keyword When Return Value Mismatch
 
 Retry Double Keywords When Error
     [Arguments]  ${keyword1}  ${element1}  ${keyword2}  ${element2}  ${DoAssert}=${true}
-    :For  ${n}  IN RANGE  1  5
+    :For  ${n}  IN RANGE  1  3
     \    Log To Console  Trying ${keyword1} and ${keyword2} ${n} times ...
     \    ${out1}  Run Keyword And Ignore Error  ${keyword1}  ${element1}
     \    Capture Page Screenshot

tests/robot-cases/Group0-BAT/API_DB.robot

@@ -67,3 +67,5 @@ Test Case - Scan Image
     Harbor API Test  ./tests/apitests/python/test_scan_image_artifact.py
 Test Case - Scan All Images
     Harbor API Test  ./tests/apitests/python/test_system_level_scan_all.py
+Test Case - Registry API
+    Harbor API Test  ./tests/apitests/python/test_registry_api.py

tests/robot-cases/Group1-Nightly/Common.robot

@@ -436,51 +436,14 @@ Test Case - Copy A Image
 Test Case - Create An New Project With Quotas Set
     Init Chrome Driver
     ${d}=  Get Current Date  result_format=%m%s
-    ${count_quota}=  Set Variable  1234
     ${storage_quota}=  Set Variable  600
     ${storage_quota_unit}=  Set Variable  GB
     Sign In Harbor  ${HARBOR_URL}  ${HARBOR_ADMIN}  ${HARBOR_PASSWORD}
-    Create An New Project    project${d}  count_quota=${count_quota}  storage_quota=${storage_quota}  storage_quota_unit=${storage_quota_unit}
+    Create An New Project    project${d}  storage_quota=${storage_quota}  storage_quota_unit=${storage_quota_unit}
-    ${count_quota_ret}=  Get Project Count Quota Text From Project Quotas List  project${d}
-    Should Be Equal As Strings  ${count_quota_ret}  0 of ${count_quota}
     ${storage_quota_ret}=  Get Project Storage Quota Text From Project Quotas List  project${d}
     Should Be Equal As Strings  ${storage_quota_ret}  0Byte of ${storage_quota}${storage_quota_unit}
     Close Browser
 
-Test Case - Project Image And Chart Artifact Count Quotas Dispaly And Control
-    Init Chrome Driver
-    ${d}=  Get Current Date  result_format=%m%s
-    ${count_quota}=  Set Variable  2
-    ${storage_quota}=  Set Variable  500
-    ${storage_quota_unit}=  Set Variable  MB
-    ${image}=  Set Variable  redis
-    #For docker-hub registry
-    #${sha256}=  Set Variable  9755880356c4ced4ff7745bafe620f0b63dd17747caedba72504ef7bac882089
-    #For internal CPE harbor registry
-    ${sha256}=  Set Variable  0e67625224c1da47cb3270e7a861a83e332f708d3d89dde0cbed432c94824d9a
-
-    ${image_size}=    Set Variable    34.14MB
-    Sign In Harbor  ${HARBOR_URL}  ${HARBOR_ADMIN}  ${HARBOR_PASSWORD}
-    Create An New Project  project${d}  count_quota=${count_quota}  storage_quota=${storage_quota}  storage_quota_unit=${storage_quota_unit}
-    Push Image  ${ip}  ${HARBOR_ADMIN}  ${HARBOR_PASSWORD}  project${d}  ${image}  sha256=${sha256}
-    ${count_quota_ret}=  Get Project Count Quota Text From Project Quotas List  project${d}
-    Should Be Equal As Strings  ${count_quota_ret}  1 of ${count_quota}
-    ${storage_quota_ret}=  Get Project Storage Quota Text From Project Quotas List  project${d}
-    Should Be Equal As Strings  ${storage_quota_ret}  ${image_size} of ${storage_quota}${storage_quota_unit}
-    #Push Image  ${ip}  ${HARBOR_ADMIN}  ${HARBOR_PASSWORD}  project${d}  alpine
-    Go Into Project  project${d}
-    Switch To Project Charts
-    Upload Chart files
-    Cannot Push image  ${ip}  ${HARBOR_ADMIN}  ${HARBOR_PASSWORD}  project${d}  busybox  err_msg=Quota exceeded when processing the request of adding 1 of count resource, which when updated to current usage of 2 will exceed the configured upper limit of 2
-    ${count_quota_ret}=  Get Project Count Quota Text From Project Quotas List  project${d}
-    Should Be Equal As Strings  ${count_quota_ret}  2 of ${count_quota}
-    Go Into Project  project${d}
-    Delete Repo  project${d}/${image}
-    ${count_quota_ret}=  Get Project Count Quota Text From Project Quotas List  project${d}
-    Should Be Equal As Strings  ${count_quota_ret}  1 of ${count_quota}
-    Push image  ${ip}  ${HARBOR_ADMIN}  ${HARBOR_PASSWORD}  project${d}  busybox
-    Close Browser
-
 # Make sure image logstash was pushed to harbor for the 1st time, so GC will delete it.
 Test Case - Project Quotas Control Under GC
     Init Chrome Driver
@@ -533,14 +496,15 @@ Test Case - Project Storage Quotas Dispaly And Control
 Test Case - Project Quotas Control Under Copy
     Init Chrome Driver
     ${d}=  Get Current Date  result_format=%m%s
-    ${count_quota}=  Set Variable  1
     ${image_a}=  Set Variable  redis
     ${image_b}=  Set Variable  logstash
-    ${image_a_ver}=  Set Variable  5.0
+    ${image_a_ver}=  Set Variable  donotremove5.0
-    ${image_b_ver}=  Set Variable  6.8.3
+    ${image_b_ver}=  Set Variable  do_not_remove_6.8.3
+    ${storage_quota}=  Set Variable  330
+    ${storage_quota_unit}=  Set Variable  MB
     Sign In Harbor  ${HARBOR_URL}  ${HARBOR_ADMIN}  ${HARBOR_PASSWORD}
     Create An New Project  project_a_${d}
-    Create An New Project  project_b_${d}  count_quota=${count_quota}
+    Create An New Project  project_b_${d}  storage_quota=${storage_quota}  storage_quota_unit=${storage_quota_unit}
     Push Image With Tag  ${ip}  ${HARBOR_ADMIN}  ${HARBOR_PASSWORD}  project_a_${d}  ${image_a}  tag=${image_a_ver}  tag1=${image_a_ver}
     Push Image With Tag  ${ip}  ${HARBOR_ADMIN}  ${HARBOR_PASSWORD}  project_a_${d}  ${image_b}  tag=${image_b_ver}  tag1=${image_b_ver}
     Go Into Project  project_a_${d}
@@ -625,44 +589,6 @@ Test Case - Robot Account
     Push image  ${ip}  robot${d}  ${token}  project${d}  hello-world:latest  is_robot=${true}
     Pull image  ${ip}  robot${d}  ${token}  project${d}  hello-world:latest  is_robot=${true}
 
-Test Case - Read Only Mode
-    Init Chrome Driver
-    ${d}=   Get Current Date    result_format=%m%s
-    Sign In Harbor  ${HARBOR_URL}  ${HARBOR_ADMIN}  ${HARBOR_PASSWORD}
-    Create An New Project  project${d}
-
-    Enable Read Only
-    Cannot Push image  ${ip}  ${HARBOR_ADMIN}  ${HARBOR_PASSWORD}  project${d}  busybox:latest
-
-    Disable Read Only
-    Sleep  5
-    Push image  ${ip}  ${HARBOR_ADMIN}  ${HARBOR_PASSWORD}  project${d}  busybox:latest
-    Close Browser
-
-Test Case - Can Not Copy Image In ReadOnly Mode
-    Init Chrome Driver
-    ${random_num1}=   Get Current Date    result_format=%m%s
-    ${random_num2}=   Evaluate  str(random.randint(1000,9999))  modules=random
-
-    Sign In Harbor  ${HARBOR_URL}  ${HARBOR_ADMIN}  ${HARBOR_PASSWORD}
-    Create An New Project  project${random_num1}
-    Create An New Project  project${random_num1}${random_num2}
-
-    Go Into Project  project${random_num1}  has_image=${false}
-    Sleep  1
-    Push Image With Tag  ${ip}  ${HARBOR_ADMIN}  ${HARBOR_PASSWORD}  project${random_num1}  redis  ${image_tag}
-    Sleep  1
-    Enable Read Only
-    Go Into Repo  project${random_num1}/redis
-    Copy Image  ${image_tag}  project${random_num1}${random_num2}  ${target_image_name}
-    Retry Wait Element Not Visible  ${repo_retag_confirm_dlg}
-    Navigate To Projects
-    Go Into Project  project${random_num1}${random_num2}  has_image=${false}
-    Sleep  10
-    Go Into Project  project${random_num1}${random_num2}  has_image=${false}
-    Disable Read Only
-    Close Browser
-
 Test Case - Push Docker Manifest Index and Display
     Init Chrome Driver
     ${d}=    Get Current Date    result_format=%m%s
@@ -695,6 +621,7 @@ Test Case - Push Docker Manifest Index and Display
     Go Into Repo  test${d}/index${d}
     Retry Double Keywords When Error  Go Into Index  index_name=${null}  Page Should Contain Element  ${tag_table_column_os_arch}
     Page Should Contain Element  ${artifact_rows}  limit=2
+    Close Browser
 
 Test Case - Push CNAB Bundle and Display
     Init Chrome Driver
@@ -716,6 +643,7 @@ Test Case - Push CNAB Bundle and Display
     Go Into Repo  test${d}/cnab${d}
     Retry Double Keywords When Error  Go Into Index  index_name=${null}  Page Should Contain Element  ${tag_table_column_os_arch}
     Page Should Contain Element  ${artifact_rows}  limit=3
+    Close Browser
 
 Test Case - Push Helm Chart and Display
     Init Chrome Driver
@@ -738,3 +666,43 @@ Test Case - Push Helm Chart and Display
     Go Into Project  test${d}
     Wait Until Page Contains  test${d}/${repo_name}
     Retry Double Keywords When Error  Go Into Repo  test${d}/${repo_name}  Page Should Contain Element  ${tag_table_column_vulnerabilities}
+    Close Browser
+
+Test Case - Can Not Copy Image In ReadOnly Mode
+    Init Chrome Driver
+    ${random_num1}=   Get Current Date    result_format=%m%s
+    ${random_num2}=   Evaluate  str(random.randint(1000,9999))  modules=random
+
+    Sign In Harbor  ${HARBOR_URL}  ${HARBOR_ADMIN}  ${HARBOR_PASSWORD}
+    Create An New Project  project${random_num1}
+    Create An New Project  project${random_num1}${random_num2}
+
+    Go Into Project  project${random_num1}  has_image=${false}
+    Sleep  1
+    Push Image With Tag  ${ip}  ${HARBOR_ADMIN}  ${HARBOR_PASSWORD}  project${random_num1}  redis  ${image_tag}
+    Sleep  1
+    Enable Read Only
+    Go Into Repo  project${random_num1}/redis
+    Copy Image  ${image_tag}  project${random_num1}${random_num2}  ${target_image_name}
+    Retry Wait Element Not Visible  ${repo_retag_confirm_dlg}
+    Navigate To Projects
+    Go Into Project  project${random_num1}${random_num2}  has_image=${false}
+    Sleep  10
+    Go Into Project  project${random_num1}${random_num2}  has_image=${false}
+    Disable Read Only
+    Sleep  10
+    Close Browser
+
+Test Case - Read Only Mode
+    Init Chrome Driver
+    ${d}=   Get Current Date    result_format=%m%s
+    Sign In Harbor  ${HARBOR_URL}  ${HARBOR_ADMIN}  ${HARBOR_PASSWORD}
+    Create An New Project  project${d}
+
+    Enable Read Only
+    Cannot Push image  ${ip}  ${HARBOR_ADMIN}  ${HARBOR_PASSWORD}  project${d}  busybox:latest
+
+    Disable Read Only
+    Sleep  5
+    Push image  ${ip}  ${HARBOR_ADMIN}  ${HARBOR_PASSWORD}  project${d}  busybox:latest
+    Close Browser

tests/robot-cases/Group1-Nightly/Upgrade.robot

@@ -27,7 +27,7 @@ Test Case - Manage project publicity
     Body Of Manage project publicity
 
 Test Case - Scan A Tag In The Repo
-    Body Of Scan A Tag In The Repo
+    Body Of Scan A Tag In The Repo Use Trivy
 
 Test Case - List Helm Charts
     Body Of List Helm Charts

tests/robot-cases/Group1-Nightly/multi_scanners.robot

@@ -0,0 +1,29 @@
+
+// Copyright (c) 2017 VMware, Inc. All Rights Reserved.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//    http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+*** Settings ***
+Documentation  Harbor BATs
+Resource  ../../resources/Util.robot
+Default Tags  Nightly
+
+*** Variables ***
+${HARBOR_URL}  https://${ip}
+${SSH_USER}  root
+${HARBOR_ADMIN}  admin
+
+*** Test Cases ***
+Test Case - Get Harbor Version
+#Just get harbor version and log it
+    Get Harbor Version